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Aimed  at  improving  officer  education  through  the  use  of  modern  technology,  a  two¬ 
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Objective  43-03X,  Education  and  Training.  (A  third  project,  Multi-Media  Course  Devel¬ 
opment,  was  funded  separately  by  the  U.S.  Office  of  Education  and  reported  (separately 

CAI  techniques  and  methods  utilized  in  the  dual  projects  (CAI-Teletype  and  CAI- 
1500)  are  discussed  under  three  categories:  computational,  non-computational ,  and 
computer  management  of  instruction. 

Courses  in  science,  engineering,  and  naval  science  were  developed  and  tested  in 
the  CAI-Teletype  Project  which  then  went  operational  with  125  teletype  terminals 
accessing  a  Honeywell  (GE)  635  computer  system. 

The  CAI- 1500  Project  featured  sophisticated  student  terminals  and  utilized  the 
IBM  1500  Instructional  System  in  a  more  extensive,  more  tutorial  manner  aimed  at 
replacing  parts  or  all  of  conventional  instructional  in  selected  situations.  Instruc¬ 
tional  modules  were  developed  for  four  academic  courses:  General  Chemistry,  Modern 
Physics,  Naval  Operations  Analysis,  and  Basic  Russian  I. 
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FOREWORD 

Required  levels  of  personnel  readiness  and  capability  in  the  Navy  are 
dependent  upon  education  and  training  programs.  To  achieve  improvements 
in  the  Navy's  education  and  training  program,  the  Chief  of  Naval  Operations 
established  Advanced  Development  Objective  43-03X,  Education  and  Training. 

A  stated  objective  of  the  ADO  was  to  "test  the  feasibility  of  available 
new  advances  in  training  technology  ...  as  a  means  for  providing  and 
maintaining  increased  personnel  capabilities."  Amplifying  data  contained 
in  the  basic  ADO  document  identified  a  number  of  advances,  already  achieved 
under  the  research  and  exploratory  development  programs,  sponsored  by  the 
Department  of  Defense  and  developed  by  various  institutions,  that  were 
judged  to  be  ready  for  feasibility  testing  under  the  advanced  development 
concept.  One  of  the  technologies  so  identified  was  the  "use  of  computer- 
aided  and  other  automated  techniques  in  instructional  procedures  and  content." 

Following  the  establishment  of  ADO  43-03X  on  6  April  1966,  several  efforts 
in  computer-assisted  instruction  were  proposed.  Because  the  ADO  designated 
"Career  Development  Training  for  Line  Officer"  as  a  high  priority  substantive 
training  area,  a  CAI  effort  was  initiated  at  the  United  States  Naval  Academy, 
focusing  on  the  development  and  utilization  of  computer-assisted  instruction 
concepts  and  systems  to  improve  the  effectiveness  of  Navy  officer  education. 

Another  high  priority  area  identified  in  the  ADO  was  "Basic  Electronics 
Training."  An  effort  underway  at  the  Naval  Personnel  and  Training  Research 
Laboratory,  San  Diego,  is  responsive  to  this  requirement. 

It  is  expected  that  these  two  projects  will  provide  the  information 
necessary  for  determining  whether  the  systems  and  technologies  evaluated 
should  be  accepted  by  the  Navy  for  use  in  officer  education. 
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I .  INTRODUCTION 


A.  NAVY  EDUCATION  AND  TRAINING 

1.  PURPOSE 

Navy  training  supports  the  Navy's  manpower  and  personnel  manage¬ 
ment  system  —  an  adaptive  system,  responsive  to  changing  qualitative/ 
quantitative  demands  for  distribution  and  utilization,  and  to  changing 
availabilities  for  input  from  different  categories  of  manpower  resources. 
Navy  education  and  training,  as  a  subsystem  of  the  manpower  and  personnel 
management  system,  is  expected  to  develop  the  capabilities  of  both  on¬ 
board  and  newly  procured  personnel  to  meet  the  qualification  requirements 
in  widely  diversified  functional  or  occupational  areas;  in  different 
degrees  of  specialization;  and  at  all  levels  of  proficiency. 

2.  NAVAL  OFFICER  EDUCATION 

Each  year  the  Navy  commissions  about  12,000  new  officers  to 
maintain  the  Navy  to  meet  its  present  world-wide  commitments ,  and  to 
provide  the  experienced  leaders  needed  for  the  Navy  of  tomorrow.  Since 
the  maximum  number  of  officers  the  Navy  may  have  is  set  by  law,  the  number 
of  new  officers  needed  is  determined  by  annual  losses  expected  from  retire¬ 
ments,  releases  from  active  duty,  and  similar  attrition. 

The  Navy  has  a  variety  of  programs  to  obtain  new  officers.  They 
can  be  divided  into  three  categories:  long,  medium,  and  short  range. 
Long-range  programs  include  the  Naval  Academy  (USNA) ;  the  Naval  Reserve 
Officers  Training  Corps  CNROTC) ,  both  Regular  and  Four-year  Contract; 
and  the  Navy  Enlisted  Scientific  Education  Program  (NESEP) .  With  the 
exception  of  the  NROTC  Four-year  Contract  program,  these  programs  are 
designed  to  produce  career  officers,  and  their  graduates  are  commissioned 
as  Regular  officers. 
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Medium- range  programs,  which  are  designed  to  produce  Reserve 
Officers,  include  the  NROTC  Two-year  Contract  Program,  the  Reserve 
Officer  Candidate  Program  (ROC),  the  Aviation  Reserve  Officer  Candidate 
Program  (AVROC) ,  and  the  College  Junior  Program  for  women  officer  can¬ 
didates  . 

Short-range  programs  include  the  Officer  Candidate  School  (OCS) , 
Women  Officers  School  (WOS) ,  Aviation  Officer  Candidate  School  (AOC) ,  and 
Naval  Aviation  Officer  Candidate  School  (NAOC) .  These  programs  produce 
Reserve  officers,  and  all  require  that  the  candidates  have  baccalaureate 
degrees  prior  to  enrollment.  Since  these  programs  are  of  relatively 
short  duration,  they  can  be  expanded  or  contracted  rapidly  to  meet 
changing  requirements  for  newly- commissioned  officers  that  result  from 
changes  in  the  world  situation,  the  economy,  and  other  factors.  Hence, 
they  are  "safety  valves"  in  officer  procurement,  that  may  be  opened  or 
closed  quickly  as  conditions  require. 

3.  IMPROVEMENT  IN  OFFICER  EDUCATION 

Technical  advances  have  increased  the  complexity  of  the  Navy's 
operational  systems  to  such  an  extent  that  the  command  function  has  become 
correspondingly  difficult.  In  order  to  meet  this  need  for  greater 
understanding,  the  naval  officer  requires  a  broad  and  comprehensive 
background  with  a  firm  foundation  in  mathematics,  sciences,  and 
engineering.  Although  the  scope  of  knowledge  to  be  mastered  by  the 
naval  officer  keeps  increasing,  the  Navy  must  hold  the  line  on  the 
length  of  time  and  costs  of  naval  officer  education.  The  Navy  anticipates 
that  through  its  investment  in  advanced  development  research  in  education 
and  training  it  can  find  techniques  and  methods  that  will  improve  the 
efficiency  of  its  officer  education  programs. 

The  most  commonly  used  instructional  method  in  officer  education, 
as  in  most  college- level  institutions,  has  been  the  traditional 


professor- lecture  method.  Though  highly  successful  under  some  circum¬ 
stances,  it  is  generally  recognized  to  have  a  number  of  limitations, 
including  the  following: 

•  It  does  not  allow  the  student  to  proceed  at  his  own  individual 
rate  of  learning. 

•  It  does  not  take  into  account  individual  differences  in  ability. 

•  It  limits  the  practice  that  can  reasonably  be  devoted  to  solution 
of  complex  problems. 

•  It  is  not  conducive  to  course  revision  and  updating  because  it 
gives  little  feedback  as  to  the  effectiveness  of  the  material 
presented. 

B.  EDUCATION  AT  THE  U.  S.  NAVAL  ACADEMY 

The  United  States  Naval  Academy  is  the  undergraduate  college  of 
the  U.  S.  Navy.  Its  primary  mission  is  to  prepare  young  men  for 
professional  careers  as  officers  in  the  U.  S.  Navy  or  the  U.  S. 

Marine  Corps . 

Responsibility  for  directing  the  Naval  Academy  is  vested  in  the 
Superintendent.  He  is  assisted  by  the  Commandant  of  Midshipmen  and 
the  Academic  Dean.  The  Commandant  is  responsible  for  directing  the 
military  and  physical  training  of  midshipmen,  while  the  civilian  Dean 
supervises  the  academic  curriculum  and  academic  standards.  The  academic 
organization  consists  of  five  Academic  Divisions,  each  headed  by  a  Navy 
captain  who  reports  directly  to  the  Dean.  The  Divisions,  in  turn,  are 
subdivided  into  the  following  19  academic  departments,  the  chairmen  of 
which  may  be  either  a  civilian  professor  or  a  Navy  or  Marine  Corps  officer: 

•  Division  of  Engineering  and  Weapons 

Aerospace  Engineering 
Electrical  Engineering 
Mechanical  Engineering 
Naval  Systems  Engineering 
Weapons  and  Systems  Engineering 


3 


Division  of  Mathematics  and  Science 


Chemistry 
Computer  Science 
Environmental  Science 
Mathematics 
Physics 

•  Division  of  Naval  Command  arid  Management 

Behavioral  Science 
Management  Science 
Navigation 

Semanship  and  Tactics 

•  Division  of  U.  S.  and  International  Studies 

Are a- Language  Studies 
Economics 
Political  Science 

•  Division  of  English  and  History 

English 

History 

In  support  of  the  primary  mission,  the  basic  portion  of  the  curriculum 
has  been  developed  to  provide  each  midshipman  with  skills  and  knowledge 
necessary  for  the  performance  of  his  duties  as  a  junior  officer.  This 
requirement  is  met  through  a  sequence  of  professional/academic  courses 
and  two  summer  at-sea  training  cruises.  Emphasis  is  placed  on  seamanship, 
navigation,  engineering,  and  weaponry.  Another  portion  of  the  basic 
curriculum  provides  all  midshipmen  with  a  broad  liberal  education  in 
social  science  and  humanities  with  a  fundamental  background  in  mathematics, 
science,  and  engineering.  A  final  portion  of  the  curriculum  is  devoted 
to  an  elected  major  that  provides  the  opportunity  for  in-depth  study  in  a 
field  of  interest. 

The  26  majors  offered  include  aerospace  engineering,  electrical 
engineering,  mechanical  engineering,  marine  engineering,  naval  architecture, 
ocean  engineering,  systems  engineering,  general  engineering,  mathematics. 


chemistry,  oceanography,  physical  science,  bioscience,  physics,  analytical 
management,  operations  analysis,  European  studies  (French,  German),  Far 
Eastern  studies,  Latin  American  studies,  Soviet  studies,  economics, 

American  political  systems,  international  security  affairs,  English,  and 
history. 

Under  the  Academy's  Trident  Scholar  Program,  a  limited  number  of 
exceptional  students  are  selected  to  pursue  independent  study  and 
research  projects  of  their  choosing  during  their  senior  years.  Each 
scholar  has  a  faculty  advisor.  Scholars  carry  a  reduced  number  of  courses, 
and  their  research  and  thesis  constitute  the  major  part  of  their  academic 
program  for  the  year. 

In  order  that  a  midshipman  qualify  for  graduation  he  must  satisfactorily 
complete  the  required  professional  courses  and  other  specifically  required 
courses,  as  well  as  the  general  distribution  requirements  in  humanities/ 
social  science,  mathematics,  science,  modem  languages,  and  computer  science, 
and  finally  the  major  course  requirements.  He  must  satisfactorily  complete, 
or  validate,  a  minimum  of  140  semester  hours  of  acceptable  academic  courses 
and  achieve  a  cumulative  Quality  Point  Rating  of  at  least  2.0. 

In  addition  to  meeting  the  academic  requirements  for  the  degree,  a 
midshipman  must  meet  required  standards  of  performance  in  the  military 
and  professional  area,  including  at-sea  training,  aptitude  for  the 
service,  conduct,  and  physical  education. 

Upon  the  satisfactory  completion  of  all  requirements,  graduates  of 
the  four-year  program  are  awarded  the  Bachelor  of  Science  degree  and 
if  in  an  engineering  discipline,  the  degree  is  so  designated--for  example 
Bachelor  of  Science  in  Aerospace  Engineering. 

The  overall  academic  program  is  supported  by  approximately  500  course 
offerings  of  which  slightly  more  than  200  are  offered  each  semester, 
including  some  of  graduate  level.  In  addition,  the  academic  program 
for  the  4200  midshipmen  is  supported  by  a  faculty  of  540.  Military 
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and  ciyilian  members  of  the  faculty  are  assigned  to  academic  departments 
in  accordance  with  their  individual  backgrounds  and  talents.  For  example, 
the  Seamanship  and  Tactics  Department  is  staffed  by  officers,  whereas 
the  Area- Language  Studies  is  largely  civilian  faculty.  The  Academy's 
balanced  officer  and  civilian  faculty  is  unique  to  the  service  academies. 
The  officers,  rotated  at  intervals  of  two  and  three  years,  provide  a 
continuing  input  of  new  ideas  and  experience  from  the  Fleet.  The  civilians 
provide  a  core  of  professional  scholarship  and  teaching  experience  as  well 
as  continuity  to  the  Academy's  educational  program. 

Although  an  average  class  size  is  20  students,  there  are  courses  which 
lend  themselves  to  large  sections  of  60  to  80  students,  while  a  number  of 
elective  courses  have  an  enrollment  of  only  10-15  students. 

It  is  interesting  to  note  that,  although  the  faculty  devotes  a  consider¬ 
able  amount  of  time  to  the  academic  counseling  and  individualized  tutoring 
of  midshipmen,  especially  those  in  academic  difficulty,  the  attrition  rate, 
which  is  generally  near  30  percent,  is  like  that  at  many  other  academic 
institutions . 

The  1300  young  men  who  are  admitted  to  the  plebe  class  each  year  come 
from  every  state  in  the  Union  and  from  backgrounds  reflecting  every  facet 
of  American  life.  Each  candidate  must  meet  general  eligibility  and 
physical  requirements.  In  the  final  selection  of  candidates  who  are 
competing  for  appointments,  the  Naval  Academy  takes  into  account  the 
quality  of  the  secondary  school  record,  additional  study,  the  College 
Board  or  ACT  scores,  extracurricular  activities,  athletics,  honors  and 
awards,  employment  outside  school  hours,  recommendations  of  teachers, 
guidance  counselors,  principals,  coaches  and  others  who  know  first-hand 
the  candidate^  accomplishments,  his  potential,  and  his  motivation  for  a 
career  in  the  Naval  Service. 
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C.  EDUCATIONAL  TECHNOLOGY  PROJECTS  AT  THE  NAVAL  ACADEMY 


Three  projects  --  all  aimed  at  improving  officer  education  through 
the  use  of  modern  technology  --  have  been  conducted  at  the  Naval  Academy 
by  the  Academy’s  Educational  Systems  Center.  They  are  designated  as: 

The  CAI -Teletype  Project;  The  CAI-1500  Project;  and  The  Multi-Media 
Course  Development  Project. 

The  first  two  projects  were  initiated  in  1967  and  supported  by  the  Bureau 
of  Naval  Personnel  under  the  ADO  43-03X  program.  The  Multi-Media  Project 
was  funded  by  the  U.  S.  Office  of  Education. 

This  report  is  concerned  only  with  the  two  CAI  projects  (CAI-Teletype 
and  CAI-1500)  conducted  under  ADO  43-03X.  These  two  projects  concentrated 
on  developing  and  evaluating  the  use  of  computers  as  an  instructional  tool 
--  computer- ass is ted  instruction,  or  CAI. 

The  goals  of  this  effort  were  to  alleviate  some  of  the  problems, 
enumerated  earlier  in  this  chapter,  by  demonstrating  the  use  of  the 
computer  to: 

•  Individualize  instruction 

•  Facilitate  the  solution  of  complex  problems 

•  Aid  in  determining  optimum  course  content 

•  Record  and  analyze  student  performance. 

(The  Multi-Media  Project  also  included  some  elements  of  computer- as sis ted 
instruction.  Information  on  that  project  is  available  through  ERIC  -- 
Educational  Resources  Information  Center,  U.  S.  Department  of  Health, 
Education,  and  Welfare.) 

D.  CAI  TECHNIQUES  AND  METHODS 

In  the  context  of  this  project.  Computer- Assisted  Instruction  is 
defined  as  the  utilization  of  computers  as  an  integral  part  of  the 
learning  process.  While  it  is  useful  to  distinguish  between  student- 
authored  and  professor-authored  programs  and  between  tutorial  and  non¬ 
tutorial  techniques  (such  as  simulation,  gaming,  data  reduction),  all 
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of  these  are  viewed  as  being  within  the  domain  of  CAI .  CAI  techniques 
and  methods  are  discussed  under  three  categories:  (1)  computational, 

(2)  non-computational,  and  (3)  computer  management  of  instruction. 

1.  COMPUTATIONAL  TECHNIQUES 

a.  Simulation 

Computer  simulation  of  laboratory  exercises  and  other  real 
world  situations  represents  a  significant  CAI  technique.  In  this  situation 
a  mathematical  model  is  generated  which  simulates  the  real  life  occurrence 
of  a  particular  environment  and  allows  the  student  to  interact  with  the 
model.  A  simulation  can  also  be  a  group  classroom  demonstration  which 
is  integrated  into  the  traditional  instructional  setting.  This  technique 
can  provide  learning  experiences  to  students  that  might  not  otherwise  be 
available  because  of  factors  such  as  safety,  equipment  cost  or  availability, 
prohibitive  set-up  time,  or  other  factors  of  cost  or  convenience. 

Because  of  such  constraints,  it  has  often  been  impractical  to 
give  each  student  hands-on  exposure  to  every  "system"  he  will  encounter 
in  his  studies.  Therefore  computer  simulations  make  a  very  valuable 
contribution  by  circumventing  these  constraints  and  putting  in  the 
student's  hands  the  complicated  "thing"  his  instructor  and  textbook 
have  been  endeavoring  to  describe  to  him. 

b .  Problem  Solving 

Problem-solving  refers  to  the  use  of  a  computer  to  solve 
quantitative  problems,  and  the  student  uses  a  language  like  FORTRAN 
or  BASIC  to  accomplish  his  purpose.  He  writes  a  program  and  enters 
his  data.  In  this  mode  the  computer  is  used  to  do  what  it  is  primarily 
designed  to  do.  A  problem  often  loses  its  significance  when  over¬ 
simplified  to  permit  a  solution  within  a  reasonable  period  of  time. 

Through  remote  time-sharing  and  simplified  programming  language, the 
potential  of  this  technique  as  an  educational  device  is  realized. 
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In  addition  to  allowing  for  efficient  solution  of  complex  problems  the 
remote  time-sharing  system  permits  the  student  to  learn  as  he  performs, 
since  on-line  debugging  is  accomplished  without  delay  or  loss  of 
problem  continuity. 

c.  Data  Reduction  and  Formatting 

Although  this  mode  of  CAI  might  be  considered  a  subset  of 
simulation  and/or  problem  solving,  it  is  significantly  distinct  to 
warrant  separate  discussion.  Data  reduction  and  formatting  involves 
collecting  information  in  raw  data  form  and,  after  simulation  or 
problem  solving,  formatting  the  output.  Essential  to  a  laboratory 
exercise, for  example,  is  the  student's  ability  to  perform  the  experiment 
and  accurately  collect  information  in  raw  data  form.  The  raw  data  must 
be  interpreted,  and  this  may  require  mathematical  calculations  as  well 
as  the  construction  of  graphs  or  tables  before  an  analysis  and  conclusion 
can  be  made . 

2.  NON -COMPUTATIONAL  TECHNIQUES 
a.  Drill  and  Practice 

Drill  and  practice  is  the  use  of  a  computer  to  present 
learning  materials,  such  as  language  drills  and  problems  in  arithmetic, 
which  utilize  the  same  sequence  and  format.  The  instructional  strategies 
are  usually  quite  simple,  as  the  computer  is  used  to  provide  practice  in 
particular  concepts  and  diagnose  a  student's  weaknesses.  There  are 
basically  two  types  of  drill  and  practice  exercises.  The  first  consists 
of  practice  problems  and  routines  that  are  designed  by  the  course  author, 
generated  by  the  instructional  program,  and  presented  to  the  student  via 
the  computer  terminal.  The  second  type  of  drill  and  practice  exercise  is 
substantially  different  as  the  student  is  allowed  to  generate  his  own 
problem  set.  Such  an  exercise  provides  the  student  with  the  opportunity 
to  construct  his  own  problem  set  and  input  data  and  to  respond  with  his 
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solution  to  the  practice  problem.  The  computer  calculates  the  correct 
answer  and  compares  it  to  the  student's  response.  The  instructional 
program  may  branch  the  student  to  particular  additional  problems,  to 
remedial  sequences ;  or  it  may  suggest  the  student  construct  a  particular 
kind  of  problem  or  one  of  increased  difficulty. 

b .  Tutorial 

Tutorial  instruction  is  a  level  of  instruction  that  not  only 
involves  dialogue  but  also  the  other  modes.  For  example,  the  consequences 
of  a  student's  response  to  a  question  may  be  drill  and  practice,  or  it  may 
be  a  simulation,  etc.  This  technique  is  characterized  by  (1)  the  presenta¬ 
tion  of  learning  material  to  the  student  in  various  step  sizes,  (2)  a 
question  or  test  item,  and  (3)  evaluation  of  the  student's  response, 
reinforcement,  and/or  branching  to  another  section  of  the  instructional 
program.  Essentially,  there  are  two  basic  formats  in  which  the  tutorial 
technique  is  presently  being  used  in  CAI  lesson  design.  The  first  con¬ 
sists  of  a  "complete"  tutorial  dialogue  between  the  student  and  the 
computer.  This  is  to  imply  that  all  information,  concepts,  and  learning 
materials  are  presented  to  students  via  the  CAI  terminal;  and  no  "off¬ 
line"  methods  are  used.  The  second  type  of  tutorial  lesson  is  referred 
to  as  a  "partial"  tutorial  dialogue.  The  student's  initial  exposure  to  the 
material  to  be  learned  is  from  an  off-line  source,  such  as  a  traditional 
lecture,  ETV,  or  reading  material. 

c.  On-Line  Testing 

Another  basic  CAI  technique  is  testing.  This  technique 
can  be  applied  either  as  a  stand-alone  application  or,  as  most  frequently 
occurs,  in  combination  with  one  of  the  other  basic  techniques.  There  are 
numerous  methods  of  applying  the  testing  technique  to  CAI  instructional 
strategies.  Measures  of  student  attainment  of  specific  objectives, 
diagnostic  testing  for  remedial  sequences,  curriculum  validation 
criteria,  and  administrative  testing  are  some  applications  of  the  testing 
technique . 
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3.  COMPUTER  MANAGEMENT  OF  INSTRUCTION  (CMI) 


The  use  of  computers  as  a  management  tool  is  a  broad  and 
important  topic.  The  primary  function  of  the  computer  is  to  assist 
the  administrator,  curriculum  developer,  researcher,  teacher,  and 
student  in  the  planning,  development,  and  prescription  of  instruction. 

Owing  to  the  particular  goals  and  limitations  in  this  project,  the  full 
potential  of  this  area  was  not  investigated.  Emphasis  was  placed  mainly 
on  the  management  of  student  data  for  course  revision,  development  of 
remedial  sequences,  as  well  as  data  reduction  and  reporting  for  evaluation 
of  students  and  materials. 

4.  CAI  CLASSIFICATION  PROBLEMS 

It  is  evident  from  examining  the  foregoing  discussion  of 
the  six  techniques  that  CAI  materials  generally  cannot  be  classified 
according  to  one  specific  technique  for  at  least  two  reasons: 

•  First  -  A  considerable  amount  of  quality  CAI  material  resists 
being  fitted  neatly  under  one  heading,  but  rather  has  elements  of  several. 
For  example:  an  effective  simulation  may  begin  with  a  pre-test  used 
both  for  evaluative  and  prescription  purposes;  since  unguided  discovery 
learning  is  frequently  unproductive,  certain  hints,  diagnostic  routines, 
and  tutorial  remediation  may  be  incorporated;  depending  on  the  complexity 
and  options  for  data  printout,  extensive  use  may  be  made  of  statistical 
and  graphic  routines;  and  finally,  to  insure  that  the  student  accomplishes 
the  objectives  set  for  the  simulation,  tutorial  routines  may  be  used  to 
probe  the  student's  "understanding"  in  the  context  of  the  simulation 
(which  may  involve  complex  problem  solution  where  the  student  modifies 
equations  or  sets  up  his  own  mathematical  models  for  testing) . 

•  Second  -  The  manner  in  which  the  CAI  materials  are  used  is  a 
critical  dimension  of  importance  both  for  design  and  evaluation.  A 

simulation,  for  instance,  may  be  utilized:  (1)  to  introduce  a  concept  area, 
(2)  to  introduce  a  laboratory  session,  (3)  to  replace  a  laboratory,  (4)  as 
a  classroom  demonstration,  (5)  as  part  of  a  review  package,  or  (6)  as  a 
homework  exercise. 
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E.  STAFF  ORGANIZATION 


From  late  1966  till  mid-69,  the  CAI  projects  were  carried  out  by  the 
Academic  Computing  Center.  At  that  point,  the  staff  associated  with  the 
developmental  aspects  (as  opposed  to  operational)  of  the  CAI  and  Multi- 
Media  projects  was  formed  into  a  separate  organizational  entity  titled 
the  Educational  and  Management  Systems  Center,  Shortly  thereafter 
(about  the  spring  of  1970),  the  name  was  simplified  and  changed  to 
EDUCATIONAL  SYSTEMS  CENTER  (ESC)  which  is  the  term  used  throughout 
this  report.  The  general  organizational  structure  is  depicted  in 
Figure  1 . 

F.  THE  OPERATIONAL  ENVIRONMENT  AND  CONSTRAINTS 

Brief  mention  should  be  made  of  the  constraints  experienced  in  per¬ 
forming  research  and  development  in  an  operational  environment.  Similar 
circumstances  are  faced  by  anyone  conducting  long-range  research  in  an 
ongoing,  dynamic  institution,  such  as  the  Naval  Academy,  whose  primary 
purpose  is  something  other  than  research.  Yet,  too  often,  these  factors 
are  not  anticipated  in  the  research  planning  phase  nor  are  they  given 
any  weight  in  the  interpretation  of  research  findings. 

1.  CURRICULUM  REVISIONS 

Curriculum  change  during  the  course  of  a  long-range  project  is 
almost  inevitable.  For  example,  the  majors  curriculum  was  inaugurated 
at  the  Naval  Academy  in  September  1969  for  the  three  lower  classes. 

The  Class  of  1970  had  gone  too  far  with  the  old  program  to  shift  over. 

By  September  1970  all  four  classes  were  enrolled  in  the  new  curriculum. 
This  had  a  serious  effect  on  the  CAI- 1500  Project,  particularly  in  the 
Russian  and  Physics  courses. 

2 .  FACULTY  ROTATION 

Faculty  changes  occurred,  too,  as  the  result  of  the  rotation  of 
military  personnel  serving  as  instructors,  and  the  inevitable  shifts 
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Figure  1.  STAFF  ORGANIZATION,  EDUCATIONAL  SYSTEMS  CENTER 
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in  civilian  personnel  who  change  jobs,  retire,  or  die.  In  both  the 
CAI-1500  and  CAI -Teletype  projects,  instructors  who  had  worked  closely 
with  the  projects  at  the  beginning  were  no  longer  at  the  Academy  when 
the  projects  terminated.  In  a  number  of  cases,  those  who  had  participated 
in  the  initial  orientation  sessions,  prior  to  the  actual  initiation  of  the 
CAI  projects,  were  not  the  instructors  who  worked  most  closely  with  the 
projects  during  their  later  stages.  In  the  concentrated  effort  to  complete 
the  project,  there  was  little  time  for  indoctrinating  the  newcomers,  with 
the  result  that  adequate  data  was  not  always  obtained.  For  example,  some 
instructors  neglected  to  obtain  both  pre-  and  post-test  results,  and  so 
some  course  materials  could  not  be  evaluated  adequately. 

3.  STUDENT  ENVIRONMENT 

Perhaps  even  more  important  than  the  curriculum  and  faculty  changes 
were  considerations  pertaining  to  students.  The  Naval  Academy,  with  its 
relatively  small  classes  and  individual  student  counseling,  makes  a  special 
effort  to  keep  the  attrition  rate  to  a  minimum,  to  see  that  its  highly  selected 
students  complete  the  four-year  course  successfully.  Consequently,  during 
the  life  of  the  CAI  projects,  the  directors  made  a  special  effort  to  ensure 
that  students  were  not  being  used  as  "guinea  pigs,"  and  that,  despite  their 
tight  schedules  and  heavy  academic  loads,  they  suffered  no  loss  in  time  or 
effort  as  a  result  of  their  participation  in  the  projects.  Sometimes  this 
was  carried  to  extremes,  particularly  by  instructors  who  felt  compelled  to 
provide  special  instruction  to  students  in  the  experimental  group  who, 
according  to  the  research  design,  were  to  learn  the  material  via  the 
computer  in  a  tutorial  mode. 

Despite  constraints  such  as  those  just  noted,  the  Educational 
Systems  Center  was  provided  the  necessary  support  to  conduct  these  projects, 
and  numerous  individual  members  of  the  Naval  Academy  faculty  gave  generously 
of  their  time  and  talents  (Appendix  B) .  The  Naval  Academy  CAE  Advisory 
Board  met  in  Annapolis  on  several  occasions  to  review  the  progress  of 
the  projects  (Appendix  C) . 


14  - 


G.  PROJECT  FACILITIES 


This  section  provides  an  overview  of  the  facilities  used  in  the  Naval 
Academy  CAI  projects.  For  a  more  detailed  description,  the  reader  may 
refer  to  Supplement  B  (Data  Management  Subsystem)  and  Supplement  D  (CAI 
Project  Facilities). 

1.  THE  CAI -1500  PROJECT 

Throughout  the  life  of  the  CAI-1500  Project,  Ward  Hall  housed  the 
IBM  1500/1800  system.  (See  Figure  2.)  Student  terminals  were  located  in 
an  adjoining  two-story  frame  building.  Originally,  the  first  floor  of 
this  building  provided  the  necessary  space  for  a  12-carrel  classroom 
(Figure  3),  a  Resource  Center,  a  teletype  terminal,  and  engineering  and 
duplicating  equipment.  Staff  offices  and  a  keypunch  occupied  the  second 
floor.  In  the  spring  of  1969,  a  second  12-terrainal  classroom  was  installed 
on  the  second  floor  of  the  building  and  the  staff  offices  moved  to  nearby 
Dahlgren  Hall.  The  IBM  1500  system,  including  its  maintenance  and  support 
of  all  IBM  software,  was  leased  from  IBM  by  the  Naval  Academy.  At  the 
time  of  its  installation  at  the  Academy,  there  were  relatively  few  such 
systems  available,  and  the  CAI  state-of-the-art  was  in  its  infancy. 

The  system  had  been  designed  for  instruction  and  proved  to  be  capable 
of  providing  a  wide  range  of  instructional  conditions  when  a  variety 
of  instructional  devices  were  added. 

The  system  consisted  of:  (1)  equipment  and  (2)  programming 

systems . 

a.  Equipment 

Included  here  are  an  1802  central  processing  unit  (CPU) , 
a  1442  card  read/punch,  a  1443  line  printer,  a  2310  disk  storage  unit, 
a  1502  station  control,  and  27  instructional  stations. 

Each  station  could  be  used  in  any  one  of  three  modes:  author, 
proctor,  or  student.  Devices  IBM  offers  for  station  use  include  a  1510 
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Figure  2.  "THE  MACHINE  ROOM,"  SHOWING  IBM  1802 

CENTRAL  PROCESSING  UNIT  AND  PERIPHERAL 
EQUIPMENT 
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Figure  3.  ONE  OF  TWO  CAT-1500  EXPERIMENTAL  CLASSROOMS 
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instructional  display  (with  keyboard/ light  pen)  containing  a  CRT  for 
display  of  alphabetic  and  numeric  characters  or  images;  a  1518  typewriter; 
a  1512  image  projector;  and  a  1506  audio  unit.  Figures  4  and  5  show  the 
station  and  classroom  in  use. 

b .  Programming  Systems 

These  include  (1)  an  operating  system  which  supervises  all 
operations  performed  by  the  equipment  and  certain  other  CAI  programs; 
and  (2)  the  COURSEWRITER  II  language.  These  are  supported  by  background 
application  programs  and  utility  programs  that  perform  service  functions 
such  as  printing  lists  of  performance  records  and  student  records. 

Additional  programming  systems  available  for  the  1800  version 
of  the  IBM  Instructional  System  include  the  Time-Sharing  Executive  System 
(TSX)  and  the  1800  card/paper  tape  programming  system.  While  these  did 
not  support  instruction  directly,  they  were  important  for  data  management. 

2.  THE  CAI -TELETYPE  PROJECT 

The  CAI-Teletype  Project  required  space  only  for  the  teletype 
terminals  because  time-sharing  computer  support  was  provided  by  vendors 
at  remote  locations .  The  number  of  terminals  grew  from  five  in  1966  to 
40  in  1970.  After  the  Honeywell  (GE)  635  system  was  installed  at  the 
Academy  and  became  operational  in  January  1971,  it  provided  the  necessary 
computer  support  in-house,  thereby  eliminating  the  need  for  vendors.  The 
GE-635  occupies  a  computer  room  (about  2000  square  feet)  in  Ward  Hall. 
Teletype  terminals  (currently  125)  are  located  mainly  in  special  class¬ 
rooms  in  the  same  building;  others  are  at  various  locations  at  the  Academy 

The  Honeywell  (GE)  635  system  was  specifically  configured  to 
serve  the  Naval  Academy's  academic  needs  and  to  parallel  the  Dartmouth 
system  in  order  to  operate  with  the  Dartmouth  time-sharing  software  (DTSS) 
It  handles  programs  written  in  BASIC,  FORTRAN,  ALGOL,  COBOL,  LISP,  TRAC, 
LAFF,  JOVIAL,  MIX,  SIMSCRIPT,  and  computer  languages  appropriate  to  the 
operating  system  in  use. 
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Figure  4.  STUDENT  INSTRUCTIONAL  STATION  SHOWING  THE 

1500  INSTRUCTIONAL  DISPLAY,  IMAGE  PROJECTOR, 
AND  AUDIO  UNIT 
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Figure  5.  STUDENTS  AT  CAI-1500  INSTRUCTIONAL  STATIONS 
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3.  RESOURCE  CENTER 


A  Resource  Center  was  established  in  June  1967  with  funds  from 
the  Office  of  Naval  Research  (ONR) .  Its  purpose  was  to  find  a  means  of 
providing  library  support  to  development  programs  in  Educational  Technology. 
Originally  it  focused  on  the  effective  collection  and  dissemination  of 
information  helpful  to  user  groups  at  the  Naval  Academy,  In  time,  however, 
its  functions  became  much  broader,  and  it  performed  many-faceted  service  and 
public  relations  functions.  For  example,  an  information  exchange  with  about 
85  other  educational  research  projects  throughout  the  world  was  initiated; 
a  workable  storage  and  retrieval  system  utilizing  a  time-sharing  computer 
functioned  as  a  pilot  study  of  keyword  search  in  retrieval  of  reports, 
papers,  and  other  materials  not  included  in  books  or  periodicals.  This 
Center  occupied  a  space  of  19  by  17  feet  in  a  room  adjacent  to  the  first- 
floor  CAI-1500  classroom. 
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II.  THE  CAI  TELETYPE  PROJECT 


A.  INTRODUCTION 

This  project  takes  its  name  from  the  type  of  terminal  used  by  the 
student  to  communicate  with  the  computer,  namely,  the  standard  Model  33 
(or  35)  teletypewriter,  as  shown  in  Figure  6.  A  number  of  courses  in 
science,  engineering,  and  naval  science  were  selected  for  development 
and  testing  of  the  following  CAI  techniques:  (1)  Tutorial,  (2)  Simulation, 

(3)  Drill  and  Practice,  (4)  Data  Reduction,  and  (5)  Problem  Solving. 

Because  of  such  limitations  as  program  size,  speed  of  terminal  output, 
and  the  like,  the  techniques  examined  in  this  project  were  predominantly 
computational.  Early  identification  of  successful  methods  led  to  adoption 
of  many  of  the  CAI  materials  as  an  operational  part  of  instruction,  thereby 
initiating  a  period  of  still  active  growth  of  academic  use  of  computers  at 
the  Naval  Academy  and  enthusiastic  faculty  involvement. 

The  CAI-Teletype  Project  proceeded  through  four  stages  which  have 
been  labelled:  spot-testing,  implementation,  validation,  and  evaluation. 

Each  stage  had  a  duration  of  one  year  (determined  by  the  Naval  Academy 
semester  system).  Midway  through  the  project,  transition  from  experimental 
to  operational  status  occurred  for  several  of  the  CAI  courses.  This 
transition  was  accompanied  by  a  sharp  increase  in  faculty  and  administration 
interest  in  the  use  of  computers  for  instructional  purposes. 

A  detailed  description  of  this  project  is  contained  in  Supplement  A 
(CAI-Teletype  Project  Report) . 

B.  SPOT-TESTING  STAGE 

The  potential  for  increasing  the  effectiveness  of  officer  education 
became  evident  to  faculty  members  involved  in  the  ONR- sponsored  feasibility 
and  appreciation  study.*  A  decision  was  made  to  capitalize  on  the  enthusiasm 

*U.S.  Naval  Academy  Feasibility  and  Appreciation  Study  of  Remote  Terminal 
On-Line  Computing  in  Education,  Academic  Computing  Center,  U.S.  Naval 
Academy,  Annapolis,  Maryland,  October  1966. 
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Figure  6.  ASR-33  TELETYPE  CONSOLE  WITH  A 
PAPER  TAPE  PUNCH/READER 
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and  experience  gained  during  this  study  as  early  as  possible  but  in  a  modest 
way.  Specific  situations  were  selected  in  a  number  of  courses  where  CAI 
techniques  could  be  spot-tested. 

The  spot-testing  program  was  both  primitive  and  limited,  in  that 
psychometric  scales  were  not  used  nor  were  all  possible  parts  of  the 
course  included.  Nevertheless,  it  was  considered  a  necessary  step  in 
determining  not  only  the  feasibility  of  using  remote  terminals  on-line 
for  computing  in  Naval  Academy  classrooms  but  also  assessing  the  reactions 
and  observations  of  the  faculty  to  these  spot  applications.  Other  important 
aims  were  (1)  to  explore  directions  to  be  taken  for  the  most  effective  use 
of  CAI,  (2)  to  acquire  first-hand  knowledge  of  the  most  useful  applications 
that  might  be  employed,  and  (3)  to  learn  some  of  the  drawbacks  or  hindrances 
that  might  be  overcome  in  advance  of  further  effort.  It  was  expected  also 
that  this  program  would  enhance  the  management's  ability  to  make  realistic 
time  and  cost  estimates  in  this  area. 

1 .  HYPOTHESES 

In  general,  it  was  hypothesized  that  the  use  of  remote  terminals 
in  the  classroom  and  laboratory  would  contribute  to  improvements  in  one 
or  more  of  the  following  three  areas: 

•  The  time  required  to  deal  with  certain  materials. 

•  The  depth  to  which  certain  materials  may  be  treated. 

•  The  length  or  number  of  items  or  topics  which  may  be  treated. 

These  three  areas  were  further  subdivided,  and  the  following 
specific  hypotheses  were  advanced.  Namely,  it  was  theorized  that  the 
use  of  remote  terminal  computing  in  the  classroom  and  laboratory  would: 

•  Increase  student  interest,  even  among  the  poorer  students. 

•  Increase  student  participation,  even  among  the  poorer  students. 

•  Obtain  favorable  student  reaction. 

•  Obtain  favorable  faculty  reaction. 
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•  Reduce  calculation  and  problem  solution  time,  thus  allowing 
more  formal  instruction  or  investigation  time. 

•  Allow  the  demonstration  and  manipulation  of  complex  equations 
and  difficult  concepts  which  were  heretofore  impossible  in  the 
classroom. 

•  Allow  the  assignment  of  more  complex  problems  for  the  student  to 
solve. 

•  Allow  simulation  of  laboratories  where  dangerous  situations  exist, 
continuous  data  under  adverse  conditions  is  desired,  or  repeated 
runs  with  student  control  over  parameters  is  desired. 

•  Allow  use  of  analytical  methods  and  data  reduction  which  were 
not  feasible  previously. 


2.  COURSES  AND  TECHNIQUES  SELECTED 


For  detailed  information  of  the  techniques  and  findings,  see  the 
Spot-Testing  Report*.  Briefly,  the  courses  and  types  of  techniques  utilized 
were  as  follows: 

•  Chemistry  --  Laboratory  simulation 

--  Drill  and  analysis 
--  Testing 

--  Laboratory  data  reduction 
—  Classroom  simulations  of  complex  systems 

•  Operations  Analysis  --  Classroom  simulation 

•  Engineering  Classroom  demonstrations 

--  Data  reduction 
--  Design  problems 
--  Laboratory  simulations 
--  Testing 

--  Drill  and  analysis 
--  Limited  tutorial 


*Report  on  Spot  Testing  of  Computer- Assisted  Education,  No.  PR-0767-3, 
Academic  Computing  Center,  U.  S.  Naval  Academy,  Annapolis,  Maryland, 
December  1967. 
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•  Underwater  Acoustics  --  Problem  solving  simulation 

•  Physics  --  Laboratory  simulation 

--  Laboratory  data  analysis 
--  Drill  and  analysis 
--  Limited  tutorial 

•  Linear  Systems  --  Classroom  presentation  of  dynamic  motion 

•  Weapons  and  Control  System  --  Classroom  demonstration  of 

complex  solutions 

3.  SUMMARY  OF  RESULTS 

A  matrix  of  results  showing  by  course  where  a  positive  result  was 
obtained  for  each  of  the  major  hypothesis  is  provided  in  Figure  7. 

While  the  most  frequently  used  technique  was  classroom  demonstration 
(simulation),  most  faculty  and  student  comments  indicated  that  increased 
direct  student  interaction  with  the  terminal  would  be  very  desirable.  To 
accomplish  this,  the  number  of  teletype  terminals  available  for  use  needed 
to  be  increased. 

C.  IMPLEMENTATION  STAGE 
1.  OBJECTIVES 

The  purpose  of  this  stage  was  to  capitalize  on  the  more  promising 
techniques  and  experience  gained  in  the  previous  stage.  By  expanding  the 
number  of  faculty  members /courses  and  the  quantity  of  CAI  material  and  by 
using  these  CAI  materials  with  full  classes  of  students  --  that  is,  "implement- 
ingn  CAI  into  the  formal  curriculum  --  it  was  expected  that  this  stage  could 
yield  more  specific,  objective  data  regarding  CAI  feasibility.  Additionally, 
the  implementation  stage  was  intended  to  provide  a  basis  for  a  more  formal 
and  penetrating  study  of  CAI  which  was  to  follow. 

Broad  objectives  of  the  CAI-Teletype  Project  were: 

•  To  provide  the  faculty  with  an  appreciation  for  and  a 
skill  in  the  application  of  time-sharing  computers  as 
a  learning  device. 
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Hypotheses  (See  pages  24-25  ) 

increased  student  interest 

increased  student  participation 

favorable  student  reaction 

favorable  faculty  reaction 

reduced  calculation  time  - 
more  instruction  time 

demonstration  and  manipulation 
of  complex  equations  and  concepts 

assignment  of  more  complex  problems 

use  of  analytical  methods  and  data 
reduction  not  previously  feasible 


Figure  7.  MATRIX  OF  RESULTS  BY  COURSE  OF  SPOT-TESTING 

FOR  CAI- TELETYPE 
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•  To  develop  effective  teaching  techniques  in  using  remote 
computer  terminals . 

•  To  implement  CAI  effectively  into  the  Naval  Academy  curriculum. 

The  first  two  objectives  are  really  never-ending  because  awareness 
of  the  computers  potential  continually  spreads.  However,  practically 
speaking,  these  first  two  objectives  were  met  with  respect  to  the  project 
as  pointed  out  in  the  Spot-Testing  Report.  It  is  toward  the  third  objective 
that  the  attention  of  the  implementation  stage  was  directed.  Both  faculty 
enthusiasm  and  know-how  were  available.  The  problem  was  to  determine  how  the 
remote  computer  terminal,  as  an  instrument  of  learning,  could  be  profitably 
implemented  into  the  curriculum. 

The  first  two  objectives  were  accomplished  with  minimal  facilities, 
that  is,  nine  teletype  terminals  distributed  singly  among  many  disciplines. 

This  would  not  accommodate  much  student  usage.  Therefore,  a  sub-objective 
became  the  determination  of  what  facilities  were  required  and  how  to 
maximize  terminal  availability  --  availability  to  students  in  the  classroom, 
in  the  laboratory,  and  at  unscheduled  times.  Individual  faculty  members  must 
determine  whether  valuable  classroom  time  could  profitably  be  devoted  to 
on-line  activities  or  whether  outside  classroom  hours  should  be  utilized 
instead.  The  type  of  course  being  considered  has  a  direct  effect  on  this 
determination. 

Another  sub-objective  emanated  from  this  decision,  which  is  that  the 
administrative  procedure  involving  student  use  of  teletype  terminals  must 
be  described.  Such  things  as  scheduling,  grading,  and  handling  of  hardcopy 
printout  material  must  be  understood.  Additionally,  the  instructor  must  have 
a  feeling  for  what  benefits  his  students  might  be  expected  to  derive  from  the 
CAI  aspects  of  his  course.  (Refer  to  hypotheses  listed  previously  on  page  24.) 
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2.  RESULTS  OF  THE  IMPLEMENTATION  STAGE* 


There  was  no  question  that  the  problem  solution  and  simulation 
techniques  (both  in  and  outside  class)  were  extremely  powerful.  Although 
new  and  certain  to  undergo  considerable  refinement,  these  techniques  were 
already  accepted  by  a  large  group  of  science  and  engineering  professors 
including  many  who  were  not  part  of  the  CAI-Teletype  Project.  Inasmuch 
as  a  decision  had  been  reached  to  make  these  techniques  an  operational 
part  of  many  Naval  Academy  courses  and  to  provide  operational  funding  for 
this  purpose,  further  formal  investigation  of  the  techniques  was  not 
warranted.  Moreover,  the  increase  in  scope  and  depth  of  the  courses 
using  these  CAI  techniques  was  so  great  that  experimental  comparison 
with  the  conventional  counterparts  could  not  be  justified.  A  particular 
case  was  the  Aerodynamic  Performance  Laboratory  which  became  operational 
in  1969-1970.  The  basis  for  the  decision  to  go  operational  included  the 
following  observations: 

•  Student  attitude  was  overwhelmingly  positive.  The  few  negative 
comments  regarding  computer  use  pertained  to  the  pre/post-tests 
and  to  poor  computer  system  performance. 

•  Many  students  elected  to  make  use  of  computer  (not  required)  in 
solution  of  their  term  problems. 

•  The  computer  utilization  was  much  higher  than  required:  1198 
hours  were  used,  whereas  486  hours  were  required. 

•  Laboratory  experiments  were  substantially  enriched  and  expanded. 

•  The  ease  of  learning  and  using  the  BASIC  language  greatly 
facilitated  faculty  and  student  generation  of  programs. 

(Approximately  200  hours  were  expended  by  faculty  members 
in  developing  the  CAI  programs.) 

•  The  computer  was  very  effective  in  teaching  parameterization 
and  optimization  techniques. 

Drill  and  analysis  and  tutorial  techniques  remained  suspect,  however. 

It  was  concluded  that  a  more  thorough  measurement  of  the  effects  of  these 
techniques  on  student  learning  was  necessary. 

*Report  on  the  Implementation  of  Computer  Assistance  in  Education,  No.  PR-0169-14, 
Academic  Computing  Center,  U.S.  Naval  Academy,  Annapolis,  Maryland,  September 
1968. 
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D.  EVALUATION  OF  SPECIFIC  COURSES  USING  DRILL  AND  ANALYSIS/ 
TUTORIAL  TECHNIQUES 


Three  courses  selected  from  the  implementation  stage  underwent 
critical  analysis  and  revision  based  on  student  data  which  was  collected 
the  previous  year.  The  validation  testing  of  these  CAI  courses  occurred 
during  the  academic  year  1968-69,  and  they  were  further  refined  and 
tested  according  to  a  formal  research  design  during  academic  year  1969-70. 

A  description  of  the  CAI  techniques,  the  findings,  and  the  conclusion 
pertaining  to  each  of  these  courses  may  be  found  in  Supplement  A  (CAI- 
Teletype  Project  Report) .  A  summary  of  evaluation  results  is  provided 
below. 

1.  PHYSICS 

The  CAI  group  consisted  of  15  students,  and  their  performance 
on  the  final  exam  was  compared  with  the  performance  of  237  students  who 
received  only  conventional  instruction.  The  mean  score  for  the  CAI  group 
was  32.3,  and  the  mean  score  for  the  control  group  was  35.2.  The  difference 
between  these  means  was  at  a  statistically  low  level  of  significance  (.10; 
which  means  that  this  difference  could  have  occurred  ten  times  out  of  100 
by  chance).  However,  when  the  two  groups  were  equated  with  respect  to 
several  student  aptitude  and  interest  variables,  there  was  no  statistically 
significant  difference  between  them. 

2.  ELECTRICAL  SCIENCE 

Two  separate  control  groups  were  used  in  the  evaluation  of  the 
Electrical  Science  materials.  The  first  control  group  (with  43  students) 
received  a  portion  of  the  CAI  materials  in  printed  form,  and  the  second 
control  group  (with  19  students)  received  only  the  conventional  instruction. 
There  were  38  students  in  the  CAI  group.  The  mean  score  on  the  final  exam 
for  the  CAI  group  was  55.6,  the  first  control  group's  mean  score  was  54.6, 
and  the  second  control  group's  mean  score  was  52.8.  When  these  three 

groups  were  equated  with  respect  to  variations  in  student  background  and 
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aptitudes,  no  statistically  significant  differences  were  found  among 
the  groups  on  final  examination  performance. 

3.  MECHANICS  OF  MATERIALS 

A  serious  constraint  was  unintentionally  placed  on  evaluation 
of  the  CAI  materials  for  the  Mechanics  of  Materials  course  by  the  Naval 
Academy  because  of  a  recent  curriculum  change.  As  a  result  of  this  change, 
only  eight  students  took  the  course,  so  no  control  group  taking  the  course 
concurrently  with  the  CAI  group  was  available.  Due  to  the  lack  of  a  formal 
control  group,  only  loose  comparisons  of  the  CAI  class  can  be  made  with 
general  performance  levels  of  16  students  in  prior  semesters  who  were 
taught  by  the  same  instructor  teaching  this  single  section  of  eight 
students . 

The  mean  final  exam  grade  for  the  CAI  group  was  2.50,  and  the 
group  designated  as  the  control  group  had  a  mean  final  exam  grade  of  2.69. 
As  with  the  other  two  courses,  when  various  student  aptitudes  were  equated 
across  the  CAI  and  control  groups,  no  statistically  significant  differences 
were  found. 

E.  PROBLEMS  OF  EVALUATION 

1.  COURSES  BECAME  OPERATIONAL 

Probably  the  most  important  factor  is  that  the  formal  evaluation 
was  limited  to  the  three  teletype  courses  which  did  not  change  over  from 
research  funding  to  operational  funding.  It  can  reasonably  be  assumed 
that  the  CAI  materials  developed  for  these  three  courses  were  the  least 
desirable,  as  far  as  faculty  opinion  is  concerned,  of  the  materials 
developed  for  12  courses  since  they  were  never  considered  ready  for 
operational  use.  If  this  is  the  case,  then,  this  evaluation  is  not 
representative  of  the  effectiveness  of  all  CAI-Teletype  materials  developed 
at  the  Naval  Academy. 
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2 .  HARDWARE/SOFTWARE  PROBLEMS 


A  second  factor  is  that  the  effectiveness  of  the  materials  was 
probably  reduced  considerably  by  the  many  computer  system  problems  which 
developed  with  the  commercial  time-sharing  computers  in  use.  Numerous 
"system  crashes"  were  experienced  during  the  evaluation  iteration  of  the 
programs,  especially  during  the  first  six  weeks  of  the  semester.  The 
evaluation  may  not  be  generalizable  to  the  use  of  the  same  programs  with 
a  more  reliable,  more  permanent  computer  system  such  as  that  now  aboard 
at  the  Naval  Academy.  In  the  event  of  a  system  "crash"  such  as  the  ones 
which  plagued  the  experimental  sections  throughout  the  project,  the  program 
run  was  terminated  and  the  student  had  to  start  over  at  the  beginning. 
Needless  to  say,  this  was  very  discouraging  to  students,  especially 
midshipmen,  whose  time  was  limited.  It  was  also  embarrassing  and 
demoralizing  to  the  instructors  who  assigned  these  programs  to  the 
students . 

Instructors  faced  further  system  problems  derived  from  the  use  of 
a  "primary"  and  a  "back-up"  system  using  two  separate  computer  time-sharing 
services.  Because  of  this,  programs  had  to  be  loaded  onto  both  systems  and 
files  updated  constantly  so  that  the  two  systems  were  compatible.  Also,  at 
one  point,  a  difference  in  the  compilers  of  the  two  systems  was  encountered, 
causing  some  programs  to  run  on  one  system  yet  not  run  on  the  other.  These 
problems  caused  significant  extra  expenditure  of  time  on  the  part  of  the 
instructors . 

3.  DATA  COLLECTION  RESTRICTIONS 

Another  serious  restriction  on  the  evaluation  process  was  the  lack 
of  appropriate  measurement  instruments  such  as  student  study  time  records, 
validated  student  questionnaires,  and  in  some  cases,  criterion-referenced 
tests  for  CAI  and  control  groups.  No  formal  evaluation  plan  had  been  devised 
prior  to  the  teletype  evaluation  period,  so  the  evaluation  design  had  to 
simply  make  the  best  use  of  data  which  were  available  or  could  still  be 
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gathered.  Electrical  Science  was  the  only  course  for  which  an  adequate 
control  group,  taught  by  the  same  instructors,  was  available.  The  control 
group  for  Mechanics  of  Materials  was  taught  by  the  same  instructor,  but 
this  was  during  an  earlier  semester  with  a  more  typical  assortment  of 
students  for  the  course. 

4.  STUDENT  BODY  CHARACTERISTICS/CONSIDERATIONS 

For  those  interested  in  making  generalizations  concerning  tutorial- 
drill  types  of  teletype  CAI  to  other  instructional  settings,  some  comments 
are  in  order  concerning  ’’conventional  instruction”  at  the  Naval  Academy. 

As  previously  noted,  the  Academy  enjoys  a  very  favorable  student/faculty 
ratio.  Most  classes  have  an  average  of  twenty  students,  and  few  faculty 
members  teach  more  than  four  classes.  This  allows  for  greater  attention 
to  individual  students  than  one  normally  finds  at  an  institution  of  higher 
learning.  The  midshipmen  themselves  are  highly  selected  and  form  a  well- 
motivated  and  relatively  homogeneous,  very  capable  group.  Instructional 
aids  such  as  training  films  and  video  tapes,  working  models,  and  modem 
laboratory  equipment  were  widely  available.  Altogether,  the  conditions 
for  effective  conventional  instruction  approach  optimal  levels.  Thus,  the 
value  of  CAI  as  a  supplementary  device  providing  for  individualization  of 
instruction  through  self-pacing  and  attention  to  individual  differences  may 
well  have  been  reduced  in  the  Naval  Academy  educational  environment.  In 
other  situations  where  the  s tudent/faculty  ratio  is  less  favorable  and  the 
range  of  student  abilities  and  motivation  is  greater,  tutorial-drill  CAI- 
Teletype  may  prove  to  be  of  much  greater  value  in  improving  the  level  of 
instruction.  It  is  likely  that,  given  a  population  with  greater  individual 
differences,  the  usefulness  of  the  individual  differences  approach  suggested 
for  assigning  students  to  CAI  and  non-CAI  versions  of  a  course  would  become 
even  more  effective. 


F.  CURRENT  STATUS  OF  CAI-TELETYPE  AT  THE  NAVAL  ACADEMY 


In  June  1971  a  computer  usage  questionnaire  was  distributed  to  all 
faculty  members.  Each  professor  or  instructor  was  asked  to  indicate, 
for  the  courses  he  taught,  (1)  those  which  used  remote  time-sharing 
computing  as  a  regular  part  of  the  course  (CAI  as  defined  in  this  report) 
during  the  1970-71  academic  year;  (2)  those  which  would  use  CAI  during 
the  1971-72  academic  year;  (3)  the  form  of  CAI  used;  (4)  the  enrollment; 
and  (5)  the  number  of  student  hours  on-line  per  semester.  Very  briefly, 
the  results  of  this  questionnaire  showed  that: 

•  Of  540  Naval  Academy  faculty  members,  261  replied  to  the 
questionnaire,  with  259  indicating  that  they  planned  to 
use  CAI  in  one  or  more  courses. 

•  Of  570  courses  offered  by  19  departments,  103  involved  CAI  in 
academic  year  1970-71.  This  number  is  expected  to  increase  to 
178  in  academic  year  1971-72. 

•  Omitting  courses  in  English,  History,  and  Language  Area 
Studies  where  CAI  is  least  used,  there  are  386  courses  -- 

171  of  which  will  involve  the  use  of  CAI  in  academic  year  1971-72. 

•  Connect  time  projected  for  academic  year  1971-72  totals  approximately 
170,900  hours. 

Data  from  computer  system  utilization  reports  for  October  1971  (the 
first  full  month  of  academic  year  1971-72)  indicate  that  faculty  estimates 
were  probably  low.  Of  284  courses  offered  in  the  fall  term  of  academic 
year  1971-72,  83  were  estimated  to  make  moderate  to  heavy  utilization  of 
computers  (six  or  more  connect  hours  per  student).  Utilization  data  reveals 
that  approximately  92  courses  involve  substantial  computer  use  and  an 
additional  61  are  making  casual  use  of  computers.  Moreover,  with  13,971 
connect  hours  logged  in  October,  it  is  reasonably  safe  to  assume  (since 
usage  increases  toward  the  end  of  the  semester)  that  the  projected  figure 
of  79,504  connect  hours  will  be  exceeded. 
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A  breakdown  of  CAI  by  users  (major  departmental  groupings)  and  by 
techniques  is  illustrated  in  Table  1.  Figure  8  graphically  displays 
the  percentage  of  use  by  major  department  and  the  percentage  of  use 
of  the  various  modes  of  CAI  planned  for  academic  year  1971-72. 

G.  ANALYSIS  OF  CAI  GROWTH  AT  THE  NAVAL  ACADEMY  --  FACTORS,  TRENDS, 

AND  POSSIBILITIES 

The  growth  of  CAI  (number  of  remote  terminals,  number  of  courses  using 
CAI,  number  of  connect  hours)  and  the  transition  from  research  to  operational 
funding  is  shown  in  Figure  9.  The  increase  in  faculty  members  making  use  of 
CAI  (not  shown  in  figure)  has  been  similarly  steep  (0  to  1966  to  over  260 
in  1971) .  The  rates  of  increase  in  connect  hours  and  number  of  courses  and 
professors  using  CAI  show  no  sign  yet  of  leveling  off. 
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SUMMARY  OF  25  JUNE  1971  COMPUTER  USAGE  QUESTIONNAIRE  RESULTS 
TYPE  AND  QUANTITY  OF  CAI  PLANNED  FOR  AY  1971-72 
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Figure  8.  BREAKDOWN  OF  CAT-TELETYPE  UTILIZATION  RY  USERS  AND  TECHNIQUES 

(Fall  1971) 
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Figure  9.  CAI-TELETYPE  GROWTH  AND  TRANSITION  FROM  R^D  TO  OPERATIONAL  FUNDING 
(Excludes:  batch  processing,  CAI-1500  computer  utilization 
and  analog/hybrid  computing) 
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III.  THE  CAI-1500  PROJECT 


A.  INTRODUCTION 

This  chapter  describes  the  development  and  evaluation  of  the  course 
materials  implemented  on  the  IBM  1500  Instructional  System.  For  a  more 
detailed  discussion  of  the  topics  of  course  development  and  formative, 
summative,  and  experimental  evaluation,  the  reader  is  referred  to 
Supplements  F  through  K  (see  page  ii) . 

Built  around  the  1500  system,  with  a  dedicated  1800  processor,  this 
project  featured  sophisticated  remote  terminals  at  which  students  were 
presented  instructional  materials.  Each  student  terminal  incorporated 
two  or  more  of  the  units  depicted  in  Figure  4  (page  19)  under  computer 
control  .  Management  decision  for  remediation  of  any  desired  complexity 
could  be  made  instantly;  thus  this  system  offered  full  media,  management, 
and  self-pacing  capabilities.  Whereas  the  focus  of  the  CAI-Teletype 
Project  was  on  the  computer^  computational  power,  the  1500  was  utilized 
in  a  more  extensive,  more  tutorial  manner  intended  to  replace  parts  or 
all  of  conventional  instruction  in  selected  situations.  Instructional 
modules  (or  chapters)  for  the  four  academic  courses  described  below  were 
developed  for  use  on  the  IBM  1500  system: 

•  General  Chemistry  --  a  two-semester  freshman  course  required  of 
prospective  science  majors.  An  Instructional  Decision  Model ,  developed 
during  the  project  for  implementing  the  chemistry  CAI  material  via  the 
computer  and  for  serving  as  a  model  for  the  other  courses,  should  be 
applicable  to  similar  efforts.  Gas  Laws,  Kinetics,  and  a  semimicro- 
qualitative  simulation  were  featured  in  the  CAI-1500  materials. 

•  Modern  Physics —  a  one-semester  elective  with  considerable  emphasis 
on  nuclear  power.  A  modified  tutorial  approach  was  used  in  presenting  CAI- 
1500  materials  covering  three  chapters  of  the  textbook. 
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•  Naval  Operations  Analysis  --  a  four-credit,  one-semester  elective 
featuring  military  applications  of  advanced  probability  theory.  Eight 
drill-and-practice  jnodules  were  developed  for  the  CAI-1500  system,  one 
of  which  ("Decision  Theory")  has  been  reprogrammed  for  evaluation  on  the 
Naval  Academy's  new  Honeywell  CGE)  635  computer. 

•  Basic  Russian  —  a  two-semester  introductory  course  offering  six 
credits.  CAI  materials  for  dirll  and  practice  were  developed  for  about 
one- third  of  the  course.  The  random- access  audio  and  recording  unit 
controlled  by  the  1500  system  made  possible  individualized  listening 
and  speaking  exercises  in  addition  to  the  reading  and  writing  drills. 

B.  COURSE  DEVELOPMENT 

The  growing  interest  and  increased  use  of  a  variety  of  self-instructional 
programs  over  the  past  ten  years  has  generated  a  great  deal  of  discussion  about 
the  role  of  evaluation  in  assessing  the  effectiveness  of  instructional  programs 
presented  via  programmed  instruction  (PI),  computer-assisted  instruction  (CAI), 
and  other  forms  of  teaching  machines  (Holland,  1961;  Jacobs,  1961;  Glaser, 

1963;  Hartley,  1963;  Cronbach,  1963;  Briggs  et  al,  1964,  Lumsdaine,  1965; 
Glaser,  1965;  Jacobs  et  al,  1966;  Gilbert,  1966;  Joint  Committee,  1966  (a) 

(b) ;  Tyler,  1967;  Scriven,  1967). 

The  greater  array  and  different  types  of  data  generated  through  the  use 
of  self-instructional  materials  as  compared  to  conventional  instruction  has 
been  the  focal  point  of  the  discussions  about  evaluation.  Questions  have 
arisen  as  to  what  criteria  should  be  used,  how  they  should  be  used,  and  what 
techniques  of  analysis  should  be  employed.  The  developmental  model  for  self- 
instructional  programs  has  brought  about  both  the  increased  array  of  student 
feedback  data  and  the  problems  associated  with  the  analysis  of  that  data. 
Although  it  can  be  greatly  elaborated  upon,  the  basic  model  is  shown  (page  41). 
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FEEDBACK 


Instructional  objectives  are  stated  as  descriptions  of  the  expected 
capabilities  of  the  students  after  instruction.  They  are  usually  stated 
very  specifically  as  descriptions  of  a  single  student  capability.  Criterion 
test  items  are  developed  in  relation  to  each  instructional  objective. 

Student  performance  is  assessed  much  more  frequently  in  the  self- 
instructional  program  setting  than  it  is  with  conventional  instruction. 

After  the  program  is  developed  and  implemented,  the  information  from 
the  analysis  of  the  student  data  provides  feedback  to  each  point  in  the 
developmental  model.  This  process  of  analysis  and  feedback,  called 
formative  evaluation  (Scriven,  1967;  Stolurow,  1968),  is  used  to  improve 
the  program  during  its  development.  Formative  evaluation  is  usually  oriented 
towards  the  parts  of  the  entire  program  sequence.  It  provides  a  diagnostic 
basis  for  program  revision.  Although  an  overall  assessment  of  an  instruc¬ 
tional  program  may  show  it  to  be  effective,  performance  on  some  objectives 
may  be  more  than  acceptable  while  on  others  it  may  be  quite  unacceptable. 


C.  PRODUCTION  OF  CAI-1500  MATERIALS 

Production,  as  used  here,  encompasses  the  various  processes  necessary 
for  converting  a  course  author's  specifications  and  content  into  smoothly- 
running  CAI-1500  instructional  materials.  Production,  which  is  a  complex 
and  time-consuming  process,  accounted  for  the  majority  of  the  manpower  costs 
in  the  Naval  Academy  CAI-1500  Project  and  for  a  substantial  amount  of  computer 
costs.  Thus  it  is  a  major  management  problem  and  is  discussed  from  that  point 
of  view  here.  More  detail,  as  well  as  the  technical  aspects  of  production, 
are  covered  in  Supplement  C,  Production  of  CAI  Materials. 

1 .  AUTHORING 

The  authors  were  Naval  Academy  faculty  members.  They 
were  selected  by  their  departments  to  work  in  the  field  of  CAI  primarily 
on  the  basis  of  interest  and,  to  some  extent,  on  the  basis  of  teaching 
responsibilities  and  time  available.  Background  in  the  area  of  educational 
design  was  given  little  weight  in  their  selection.  Consequently  a  high 
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degree  of  support  from  the  Technical  Support  Group  of  the  Educational 
Systems  Center  was  required.  The  composition  of  the  Technical  Support 
Group  is  shown  in  Figure  1, 

Problems  encountered  in  authoring  courses  revolved  around: 

(1)  lack  of  definition  of  course  content,  (2)  difficulty  in  writing 
acceptable  behavioral  objectives,  and  (3)  developing  criterion  test 
questions.  At  first  some  faculty  members  were  not  receptive  to  the 
procedures  proposed  by  consultants  and  programmers,  but  as  they  gained 
experience,  they  recognized  the  need  for  specificity  in  all  these  areas. 

Others  saw  the  need  and  practiced  acceptable  procedures  from  the  beginning. 
Many  of  the  problems,  however,  stemmed  from  difficulty  of  communication 
between  authors  and  programmers . 

2 .  PROGRAMMING 

After  the  instructional  material  has  been  developed,  it  must  be 
entered  into  the  system.  Primary  responsibility  for  this  aspect  of  the 
effort  rested  with  the  Technical  Support  Group.  Some  course  authors, 
however,  became  involved  with  programming,  as  well  as  with  developing 
the  course  specifications;  others  did  not  concern  themselves  with 
programming  at  all . 

It  is  difficult  to  predict  what  a  programming  load  will  be  until 
specifications  for  a  course  are  written  and,  on  that  basis,  estimates  of 
programming  time  developed.  Some  factors  affecting  the  amount  of  time 
required  are  as  follows: 

a.  Type  of  Material 

If  the  material  lends  itself  readily  to  the  use  of  macros 
(calling  of  pre-programmed  instructions  by  the  use  of  a  single  instruction) 
and  if  the  answer  analysis  is  linear  in  nature,  the  material  can  be  programmed 
rapidly.  (See  Figure  10a.)  However,  the  answer  analysis  becomes  extremely 
large  (in  proportion  to  the  amount  of  text  a  student  sees)  when  the  course 
allows  the  student  to  take  one  of  several  paths,  as  in  Figure  10b. 
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One-part  question 


C  W  UN  Answer  analysis 

C  »  Correct  Response 
W  *  Wrong  Response 
UN  »  Unanticipated  Response 


question 


answer; 


Figure  a.  SIMPLE  ANSWER  ANALYSIS 


Figure  C.  AUTHOR-DETERMINED  PATH 


Question 


Correct  response;  go  to  next  question 


Anticipated  wrong  response;  give  a 
clue  and  try  question  again 


Anticipated  wrong  response  on  second 
try;  go  to  remediation 


Unanticipated  response;  give  a  clue 
and  try  question  again 


Unanticipated  response  on  second  try;  go  to 
remediation 


Give  student  correct  answer  and  remediation 


New  question  comparable  to  original 


•  Correct  response;  go  to  next  question 


Repeat  3a-4b,  as  appropriate 


Figure  b  .  COMPLEX  ANSWER  ANALYSIS 


Figure  d.  STUDENT-DETERMINED  PATH, 


Figure  10.  VARIATIONS  IN  COURSE  STRUCTURE 
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b .  Complexity  of  Course  Structure 

The  course  flow  may  be  pre-determined  by  the  author.  The  author, 
for  example,  might  specify  a  simple  flow,  such  as,  "Ask  the  student  a  ques¬ 
tion  and  if  he  answers  correctly,  send  him  on  to  the  next  one;  if  he  answers 
incorrectly,  tell  him  the  correct  answer  and  proceed  to  the  next  question." 
(See  Figure  10c.)  On  the  other  hand,  if  the  author  decides  to  have  the  stu¬ 
dent's  need  determine  his  path,  the  routine  becomes  far  more  complicated 
and  requires  more  skill  and  time  in  programming.  (See  Figure  lQd.) 

c.  Volume 

The  volume  of  material  to  be  programmed  is  an  important  factor 
in  determining  the  length  of  time  required  for  programming. 

d.  Author's  Approach 

As  the  author  progresses  from  taking  small  blocks  of  material 
to  large  portions  of  a  course,  the  volume  of  material  becomes  critical. 

Under  a  large-scale  system,  the  students  "fan  out"  over  the  course 
material,  in  accordance  with  their  abilities,  motivation,  and  so  on. 

The  larger  the  course  the  more  rigorous  the  answer  analysis  must  be  to 
accurately  diagnose  a  student's  need. 

e .  Presentation  of  Material  to  Programmer 

The  manner  in  which  the  material  is  presented  to  the  programmer 
is  an  important  factor  in  estimating  time  requirements.  The  most  efficient 
way  is  to  present  the  material  on  display  guides,  including  labels,  enter- 
and-processing  identifier,  match  response  identifiers,  and  the  like. 


f .  Revisions 

This  time-consuming  aspect  of  programming  is  necessary  when 
there  are  subject-matter  changes  in  the  course  itself. 
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3.  PRODUCTION  SUPPORT 


a.  Seminars 

While  the  Educational  Systems  Center  was  able  to  provide 
extensive  support  to  the  course  authors  in  the  areas  of  instructional 
material  design  and  programming,  some  portions  of  the  development  phase 
required  both  talents  and  facilities  not  available  in-house.  Recognizing 
this,  the  Educational  Systems  Center  arranged  for  the  presentation  of 
informal  courses  and  seminars  by  outside  groups  for  selected  faculty 
members.  Not  all  faculty-authors  availed  themselves  of  the  opportunity, 
with  the  result  that  there  was  an  uneven  level  of  author  expertise  and  a 
great  deal  of  trial-and-error  operation  during  the  early  stages  of 
production. 

b .  Film  Strips 

A  purely  mechanical  support  area  was  the  production  of  film 
strips  for  the  IBM  1512  image  projector  which  was  used  in  all  the  CAI-1500 
courses.  This  proved  to  be  one  of  the  major  bottlenecks  in  the  program. 

It  required  a  sophisticated  photography  process  for  which  the  Naval  Academy 
was  not  equipped.  Consequently,  facilities  outside  the  Academy  had  to  be 
used  and  the  time  required  often  slowed  progress. 

Obtaining  artwork  from  rough  sketches  or  drawings  by  authors 
for  use  on  the  image  projector  was  originally  the  responsibility  of  the 
Educational  Systems  Center,  but  it  proved  more  feasible  to  have  authors 
work  directly  with  the  art  studio. 

c.  Content  Review 

In  addition  to  mechanical  services  required  during  the  production 

of  course  material,  there  was  also  the  necessity  for  content  review  of  the 

material.  Educational  Systems  Center,  in  cooperation  with  the  individual 
faculty  members,  contacted  representatives  of  the  various  departments  at 
the  Naval  Academy  as  well  as  qualified  personnel  from  other  universities 
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and  from  projects  involved  in  similar  efforts.  Project  assistance  in  the 
areas  of  programming  techniques  and  documentation  standards  was  also  provided 
by  such  outside  groups  as  Stanford  University's  and  Florida  State  University's 
data  management  personnel. 

d.  Consultants 

The  majority  of  course  material  development  and  implementation 
was  accomplished  with  Naval  Academy  talent  and  facilities.  For  the  needed 
support  in  preparing  validation  and  evaluation  plans,  two  outside  consultants 
were  obtained:  Dr.  Edgar  M.  Haverland,  of  the  Human  Resources  Research  Office, 
and  Dr.  Lawrence  M.  Stolurow  (with  his  associate.  Dr.  R.  L.  Brennan)  then  at 
Harvard  and  now  at  the  State  University  of  New  York  at  Stony  Brook.  In  the 
closing  stages  of  the  project,  LeRoy  C.  Rivers,  doctoral  candidate,  Florida 
State  University,  served  as  a  consultant  regarding  statistical  evaluations 
and  the  final  write-ups. 

4.  COMPUTER  LANGUAGE 

The  IBM  1500  system  used  the  COURSEWRITER  II  language.  The 
specifications  of  the  instruction  set  make  it  essentially  a  tutorial  type 
language.  When  other  teaching  strategies  are  used  it  is  necessary  to  work 
around  the  language  by  using  functions  or  employing  the  instructions  in  an 
unorthodox  manner.  This  makes  the  CAI  presentation  appear  to  be  a  programmed 
text.  What  is  not  seen  is  the  fact  that  branches  are  not  predicated  on  a 
single  response  but  on  possibly  all  of  the  student's  previous  responses. 

Though  COURSEWRITER  does  provide  instructions  for  carrying  out  this  logic, 
it  does  not  do  it  in  the  most  efficient  manner,  e.g.,  ten  COURSEWRITER 
instructions  are  used  to  do  what  one  Boolean  instruction  would  do. 

The  big  limitation  in  COURSEWRITER  was  its  inability  to  do  floating 
point  and  complex  arithmetic.  It  was  limited  to  handling  integers  up  to 
32768.  Thus  the  Naval  Academy,  which  is  essentially  an  engineering  school, 
had  a  CAI  system  that  could  not  handle  the  mathematics  an  engineer  uses. 
Originally,  the  Mathematics  Algorithm  Translator  (MAT)  language  was  supposed 
to  perform  this  function,  but  the  limited  storage  plus  other  features  made  it 
unacceptable.  It  was  replaced  by  Kenneth  Iverson's  APL  (A  Programming  Language), 
but  this  likewise  was  not  acceptable,  primarily  because  of  its  incompatibility 
with  COURSEWRITER.  That  is,  the  COURSEWRITER  and  APL  languages  could  not  be 
used  at  the  same  time. 
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The  programming  staff  used  imagination  in  solving  many  of  the 
problems  of  COURSEWRITER  except  for  those  with  a  computational  aspect. 

5.  PROGRAMMER- AUTHOR  COORDINATION 

Authors  often  submitted  materials  for  programming  that  were  barely 
legible.  Also,  they  did  not  always  make  their  intentions  clear  to  programmers, 
with  the  result  that  revisions  in  programmed  materials  had  to  be  made.  When 
the  author  wanted  to  make  revisions,  several  ways  were  tried.  It  was  a 
fairly  simple  matter  if  the  desired  revision  meant  adding  or  changing  a 
few  instructions.  However,  if  a  major  revision  was  necessary,  it  meant 
indicating  the  change  on  a  Display  Guide  so  that  the  programmer  would 
have  an  idea  of  how  the  author  wanted  the  material  to  appear  on  the  screen. 

Two  problems  could  enter  here.  The  listing  on  which  the  author  had  noted 
his  revisions  might  be  misplaced,  or  the  Display  Guides  might  not  be  with 
the  listing.  Thus,  the  programmer  did  not  realize  a  major  revision  was  to 
be  made.  In  addition,  after  the  author  had  checked  the  revisions,  the 
programmer  or  someone  else  might  note  that  the  EP  identifiers  (which 
identify  each  question)  were  not  correct.  In  the  process  of  correcting  the 
EP  identifier,  new  errors  could  be  put  into  the  program.  A  solution  to  all 
these  problems  was  to  standardize  the  Display  Guide  as  the  communication 
vehicle  between  the  author  and  the  programmer. 

There  were  problems  of  communication  between  authors  and  programmers. 
Yet,  coordinated  teamwork  between  these  two  groups  is  essential  for  efficient 
production  of  CAI  materials. 

A  major  difficulty  in  this  area  was  that  authors  were  faculty 
members  drawn  from  and  responsible  only  to  their  own  departments.  The 
arrangement  had  several  results: 

#  Programmer  personnel  tended  to  be  loyal  to  the  author  or  the 
course,  rather  than  to  the  Educational  Systems  Center.  Thus,  friction 
often  developed  between  the  programmer  and  his  supervisor  as  well  as 
other  programmers . 
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•  Authors  often  had  other  teaching  and  committee  assignments 
which  kept  them  from  putting  full  time  on  CAI . 

•  The  Educational  Systems  Center  could  not  enforce  compliance 
with  standards  for  submitting  material  in  usable  form  and  for  the  for¬ 
mulation  of  acceptable  behavioral  objectives  for  the  course  material. 

D.  MANAGEMENT 

1.  PLANNING 

Planning  the  number  and  types  of  personnel  required  for  the  pro¬ 
duction  of  CAI  course  materials  is  a  major  management  problem.  Efficient 
planning  cannot  be  accomplished  unless  the  types  of  material  to  be  produced, 
their  complexity,  and  volume  are  known.  Both  authors  and  computer  personnel 
are  involved  and  they  must  work  as  a  team. 

2.  PERSONNEL  SELECTION 

In  general,  the  manpower  requirements  will  not  be  the  same  for  the 
early  stage  of  developing  materials  and  for  the  operational  stage  when  the 
materials  are  used  by  students.  For  example,  the  tasks  of  system  planning 
and  design,  logic  design,  coding,  testing,  and  system  shakedown  dictate 
that  programming  manpower  will  probably  be  much  higher  during  the  developmental 
stage  than  it  will  be  during  the  operational  stage.  Conversely,  computer 
operations  personnel  will  not  be  needed  until  the  computing  hardware  is 
installed  and  will  not  need  to  be  full  strength  until  the  project  enters 
the  operational  phase. 

Essential  to  both  stages,  however,  is  a  manager  of  the  group  which  is 
responsible  for  producing  the  materials  and  getting  them  on  the  system  (in 
this  project,  called  the  Technical  Support  Group).  The  manager  must  be  able 
to  deal  with  and  evaluate  the  performance  of  course  material  experts  and 
technical  personnel  as  well  as  schedule  and  administer  the  clerical  and 
computing  system  operations.  It  is  advisable  to  acquire  an  experienced 
person  or  someone  with  closely  related  experience  (e.g.,  a  line  supervisor 
at  a  CAI  project). 
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Systems  analysts  and  programmers  made  up  the  Technical  Support 
Group.  Here  a  decision  must  be  made  between  hiring  experienced  personnel 
or  trainees.  The  salaries  for  experienced  analysts  and  programmers  will, 
of  course,  be  higher,  but  less  training  will  be  necessary.  This  approach 
may  be  especially  attractive  if  the  experience  has  been  with  the  same 
programming  language  or  system. 

The  possibility  of  hiring  trainee  programmers  because  of  their  low 
salaries  may  be  appealing  to  management.  However,  this  approach  tends  to 
overlook  the  potential  loss  in  time  and  cost  if  the  trainee  leaves  his 
position. 

The  best  approach  seems  to  be  to  hire  a  few  experienced  technical 
personnel  (programmers  and  analysts),  give  them  a  few  weeks  to  become 
familiar  with  the  new  organization,  its  mission,  the  computing  system, 
and  programming  language,  and  then  hire  trainees  and/or  minor  programmers 
depending  on  the  particular  time  frame. 

3.  TRAINING 

Training  for  both  authors  and  programmers  is  essential,  as 
mentioned  earlier  in  the  chapter.  A  Faculty  Course  in  Educational 
Technology  was  provided  by  the  General  Learning  Corporation,  covering 
such  topics  as  behavioral  objectives,  criterion  tests,  and  educational 
applications  of  the  computer. 

Shortly  before  delivery  of  the  system,  IBM  furnished  training 
personnel  and  conducted  training  for  three  different  groups  (authors, 
programmers,  and  operators)  on  the  IBM  1500  Instructional  System. 

E.  DATA  MANAGEMENT  SUBSYSTEM 

This  section  summarizes  information  on  the  data  management  subsystem 
for  the  CAI-1500  Project.  Persons  desiring  a  more  detailed  description 
should  refer  to  Supplement  B,  Data  Management  Subsystem  Report. 
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1.  APPROACH 


A  data  management  system  must  be  viewed  as  a  subsystem  of  the  overall 
instructional  system.  Therefore,  before  detailed  planning  of  the  data 
management  subsystem  (DMS)  is  undertaken,  the  total  instructional  system 
must  be  defined  precisely,  revealing  the  related  subsystems,  general 
information  flow  patterns,  and  requirements.  A  simplified  schematic 
of  a  general  instruction  system  is  shown  in  Figure  11-  This  reveals,  from 
a  data  management  perspective,  the  major  elements  and  information  flows  in 
the  system.  It  also  illustrates  the  major  functions  a  computer  can  perform. 

Once  the  total  instructional  system  has  been  defined,  the  following 
steps  must  be  taken  to  determine  the  data  management  subsystem  requirements: 

a.  Identification  of  persons  who  are  to  receive  data;  the  data 
they  will  need  and  at  what  level  of  detail,  where,  and  for  what  purpose. 

b.  Analysis  of  resources  available  for  operation  of  the  DMS, 
including  personnel  and  the  computer  system  itself. 

2.  INFORMATION  FLOW 

The  process  of  developing  the  learning  activities  and  determining 
student  paths  involves  many  decisions.  These  decisions  are  predominantly 
of  two  types:  (1)  decisions  made  in  advance  of  instruction  which  are  then 
reexamined  on  the  basis  of  data  and  (2)  decisions  which  cannot  be  made  in 
advance  but  occur  only  during  instruction.  For  example,  for  a  given 
objective,  one  may  prepare  three  differing  sets  of  remedial  information 
frames  corresponding  to  three  states  of  knowledge  that  seem  likely  to 
exist  within  a  group  of  students. 

During  instruction,  the  state  of  knowledge  for  a  given  student  is 
determined  and  the  corresponding  set  of  frames  presented  to  that  student 
Following  instructions  one  then  asks  such  questions  as:  Did  each  remedial 
set  perform  as  desired.  Was  each  of  the  three  sets  necessary,  Were  the 
three  sets  sufficient.  Was  each  student  presented  the  best  set  of  frames 
for  him,  How  does  this  method  of  teaching  the  objective  compare  with 
alternates.  What  are  the  cumulative  effects  of  this  method,  etc. 
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Figure  11.  GENERAL  INSTRUCTIONAL  SYSTEM 
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The  sources  of  data  required  for  these  decisions  are  student  performance 
data  (such  as  achievement,  learning  time,  latency),  student  attitude  data, 
instructors  attitudes,  and  student  entry  characteristics  (data  about 
students  which  is  collected  prior  to  the  start  of  the  course).  For  the 
CAI-1500  Project  the  bulk  of  this  data  was  collected  on-line,  with  some 
attitude  data  and  background  data  collected  off-line  and  inserted  into  the 
system  in  batch  mode. 

Inputs  to  the  DMS  included  student  entry  and  exit  data  (card  input) 
and  raw  student  performance  records  (tape  input).  The  items  comprising 
student  entry  and  exit  data  are  listed  in  Figure  12.  The  informational 
flow  is  shown  in  Figure  13. 

3.  RESOURCES 

Design  of  the  DMS  for  the  Naval  Academy's  CAI-1500  Project  was  an 
evolutionary  process,  involving  a  series  of  tradeoff  decisions  based  on 
estimates  of  the  costs  and  benefits  of  the  many  alternatives  that  arose 
during  the  project.  Originally,  the  IBM-1500  system,  for  reasons  of  cost 
and  availability,  was  used  for  DMS  implementation.  However,  the  system 
had  a  major  limitation  for  this  application:  during  instructional 
sessions  it  was  not  possible  to  access  any  information  on  magnetic  tape. 
Consequently,  no  information  in  student  performance  files  could  be 
retrieved,  manipulated,  and  used  as  input  to  instructional  algorithms 
curing  instruction.  This  meant  that  the  DMS  could  not  function 
simultaneously  with  instruction  or  directly  support  instruction  in 
real  time.  In  order  to  use  student  performance  characteristics 
data  in  instructional  decision  making,  the  necessary  data  had  to  be 
stored  in  the  "current"  student  response  record.  (A  limited  number  of 
counters  and  switches  are  available  for  data  storage  in  response  records.) 

The  most  significant  change  in  the  original  DMS  design  came  about  as 
a  result  of  the  installation  of  the  GE-635  time-sharing  computing  system  at 
the  Academy.  Not  only  does  the  system  have  power  and  speed  but  it  has  a 
capability  that  proved  to  be  particularly  valuable  in  the  CAI-1500  Project 
data  management:  the  GECOS  simulator  (a  standard  batch  operating  system) 
allowed  background  jobs  to  be  processed  while  the  DTSS  (provided  by 
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Figure  13.  DMS  INFORMATION  FLOW,  REQUIRED  OUTPUTS 
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Dartmouth  College  for  support  of  time-sharing)  was  operating.  This 
capability  made  it  possible  to  debug  DMS  programs  on-line  and  to  execute 
jobs  immediately. 

4.  FINAL  SYSTEM  FOR  DMS 

The  final  form  of  the  Naval  Academy  DMS  is  illustrated  in  Figure  14. 
It  is  composed  of  two  parts.  One  part  operated  on  the  IBM- 1500  system 
and  provided  administrative  and  formative  evaluation  reports.  The  second 
part  operated  on  the  GE-635  providing  the  primary  information  required 
for  research  and  summative  evaluation.  Basically,  the  dividing  line 
between  these  two  parts  placed  on  the  1500  those  programs  which  were  run 
regularly  and  frequently,  while  those  which  were  run  once  or  rarely  were 
placed  on  the  GE-635.  In  addition,  the  routines  which  were  the  more  subject 
to  change,  required  the  more  sophisticated  statistical  analysis  routines, 
involved  large  correlation  matrices,  and  large  numbers  of  types  of  data  were 
run  on  the  GE-635. 

5 .  STAFF 

Many  members  of  the  Educational  Systems  Center  (ESC)  organization 
contributed  to  DMS  development.  Specifications  for  administrative, 
formative  and  summative  evaluation  reports  were  initially  provided  by 
the  project  manager  directly  to  the  computer  systems  manager.  Later  in 
the  project  additional  summative  evaluation  requirements  and  the  research 
requirements  were  specified  by  the  research  specialist.  Change  require¬ 
ments  were  submitted  via  the  ESC  Director  but  required  close  coordination 
between  the  project  manager  and  computer  systems  manager. 
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The  level  of  personnel  involved  and  approximate  time  expended 
in  DMS  development  was  as  follows: 


Project  manager 

GS-13 

1/4  man  years 

Research  specialist 

Associate 

Professor 

1/8  man  years 

Computer  systems  manager 

GS-13 

1/2  time 

1.5  man  years 

Applications  programming 

manager 

GS-9-11 

full  time 

3  man  years 

Applications  programmer 

GS-7-9 

1/4  time 

3/4  man  years 

Operators 

GS-4-5 

1/4  time 

3/4  man  years 

The  relative  amount  of  effort  required  for  development  of  the  IBM- 1500 
and  the  GE-635  portions  of  the  DMS  was  approximately  20  to  1. 


6 .  PROBLEM  AREAS 

A  number  of  problem  areas  contributed  to  high  costs  in  developing  the 
DMS.  Among  these  were:  management  of  a  complex  staff;  differences  among  the 
four  courses  for  which  CAI  materials  were  developed;  changes  in  instructional 
models;  emergent  ideas  about  formative  evaluation  requirements;  methods  for 
establishing  identifier  fields  (the  key  to  all  select  and  sort  routines); 
the  use  of  counters  and  switches  in  student  response  records;  inquiry 
capability  permitting  students  and  instructors  to  obtain  performance 
data  on-line;  availability  of  languages  and  utility  programs;  and 
procedures  for  achieving  a  routine  process. 
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F.  IMPLEMENTATION 


1 .  TRYING  OUT  MATERIALS 

The  lack  of  available  students  to  try  out  materials  proved  to  be 
a  problem.  One  might  expect  that  midshipmen  could  be  brought  to  a  class¬ 
room  for  this  purpose  at  any  time  of  day  or  night.  However,  as  previously 
stated,  the  midshipman's  time  is  very  precious  and  is  well  accounted  for. 

It  was  thought  originally  that  some  students  might  be  available  during  the 
summer,  but  even  then  time  is  scheduled  for  summer  cruises  and  the  like. 

As  a  result,  the  Educational  System  Center  had  to  rely  on  the 
opinions  of  non-authoring  faculty  members  as  to  how  successful  the  materials 
might  be  in  teaching  students.  Thus,  a  regular  class  was  the  first  group  of 
students  to  see  the  materials. 

2.  INTRODUCING  STUDENTS  TO  THE  SYSTEM 

Several  methods  of  introducing  students  to  the  system  were  tried. 
Initially  they  were  told  something  about  what  was  available  and  then  taken 
through  a  series  of  exercises  in  which  they  learned  to  "enter"  their  answers. 
Later,  they  were  simply  told  what  to  do  and  then  put  on  the  system.  More 
recently,  the  procedure  was  to  give  them  about  fifteen  minutes  of  exercises 
and  then  start  them  into  regular  course  materials. 

3.  PROCTORING 

When  authors  participate  in  the  proctoring  function,  students  are 
tempted  to  ask  them  for  help  which  is  not  really  necessary.  It  is  diffi¬ 
cult  for  the  authors  to  refrain  from  helping  the  students  as  they  want 
them  to  get  through  the  materials  with  the  minimum  amount  of  interference 
from  mechanical  difficulties  with  the  system,  whether  hardware  or  software 
related.  In  the  initial  presentations  it  was  necessary  for  the  author  to 
be  a  proctor  so  that  he  could  observe  how  the  students  were  reacting  and 
could  help  those  who  got  into  trouble  because  of  programming  or  system  errors. 
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Eventually  it  became  easier  to  determine  the  type  of  trouble  a  student 
was  having  and  for  the  non-author  proctor  to  rectify  routine  problems. 

Authors  were  encouraged  to  remain  in  the  room  adjacent  to  the  classroom. 

Since  this  gave  the  authors  a  chance  to  analyze  data  and  do  other  tasks, 
they  were  happy  to  comply. 

4 .  SCHEDULE 

As  can  be  noted  in  Table  2,  the  development  of  the  CAI  materials 
for  the  four  courses  was  begun  at  various  times  throughout  the  project. 

When  time  permitted,  the  course  materials  were  subjected  to  review  by 
one  or  more  students  in  order  to  eliminate  obvious  errors  and  inconsistencies 
in  the  CAI  programs.  However,  in  most  cases,  the  course  materials  were 
completed  barely  in  time  for  the  start  of  classes,  and  the  initial 
implementations  of  materials  developed  in  1967  were  plagued  with  many 
problems.  The  Physics  course  was  the  most  problem-prone,  while  the 
Naval  Operations  and  Russian  materials  proceeded  with  relatively  few 
problems.  The  development  of  all  the  courses  was  hindered  in  some 
respect  by  curriculum  and  personnel  changes. 

Each  course  represented  a  slightly  different  mode  of  CAI 
presentation,  from  drill  and  practice  to  some  form  of  tutorial.  In 
addition,  different  strategies  were  employed  in  their  development. 

Over  the  history  of  the  project,  the  strategies  employed  changed 
within  some  courses.  In  the  Chemistry  course,  for  example,  the 
CAI  materials  were  initially  intended  to  provide  drill  and  practice 
on  solving  various  quantitative  aspects.  However,  in  the  summer  of 
1969  the  materials  were  reduced  in  terms  of  the  number  of  topics 
covered  and  the  remaining  topics  were  expanded  to  a  more  complete 
tutorial  format. 
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TABLE  2 


DEVELOPMENT,  IMPLEMENTATION,  AND  REVISION  TIME  FRAMES 
FOR  THE  FOUR  CAI  COURSES* 


Course 


Initially 

Developed  Implementation  Revisions 


Chemistry 

1st  Semester 

Materials 

Summer  1967 

Fall  1967 

Spring  and 

Summer  1968 

Fall  1968 

Spring  and 

Summer  1969 

Fall  1969 

Spring  and 

Summer  1970 

Fall  1970 

2nd  Semester 

Materials 

Spring  and 
Summer  1969 

Spring  1970 
Spring  1971 

Summer  1970 

Physics 

Summer  1967 

Fall  1967 

Continuous  efforts 
at  revision  and 
implementation  from 
Spring  1968  to 

Fall  1969 

Spring  1970 

Spring  1971 

Summer  and  Fall  1970 

Naval  Operations 

Summer  1968 

Fall  1968 

Spring  1969 

Fall  1969 

Summer  1969 

Spring  1970 

Summer  1970 

Fall  1970 

Spring  1971 

Russian 

1st  Semester 

Materials 

Summer  1968 

Fall  1968 

Spring  and 

Summer  1969 

Fall  1969 

Spring  and 

Summer  1970 

Fall  1970 

2nd  Semester 

Materials 

Spring  and 
Summer  1969 

Spring  1970 
Spring  1971 

Summer  1970 

*See  Supplements  H-K  for  a  complete  description  of  the  development  of 
the  four  courses.  -  60  - 


G.  REVISION  PROCEDURES 


Since  the  courses  being  developed  were  part  of  a  first  effort  in  a 
new  area,  revision  was  ineyitahle.  The  amount  of  revision  necessary  varied 
with  the  course  and,  to  a  great  extent,  depended  on  the  period  of  course 
development.  That  is,  the  earlier  the  course  was  developed,  the  greater 
likelihood  that  revisions  would  be  needed.  (For  more  detail  see  Supplement 
F  (Formative  Evaluation  of  the  CAI-1500  Project  Materials) . 

Generally  speaking, • the  revisions  were  taken  care  of  during  the 
early  months  of  writing  and  programming.  Revision  was  required,  however, 
throughout  the  life  of  the  project  as  a  result  of  other  developments. 

Among  these  were  curriculum  changes,  staff  changes,  review  of  materials 
by  persons  other  than  the  authors,  and  difficulties^revealed  by  an 
analysis  of  student  data  during  the  try-out  phase.  For  example,  a 
student  might  enter  what  was  a  correct  answer  to  him,  only  to  have 
the  computer  reject  the  answer  and  provide  him  with  unnecessary  remedial 
material.  (Regardless  of  the  care  exercised  in  the  development  of 
instructional  materials,  a  certain  amount  of  this  type  of  revision 
appears  unavoidable.) 

Some  of  the  revisions  were  hardware/software  dependent  and  therefore 
affected  all  courses.  Among  these  were  such  things  as  (1)  the  amount  of 
text  that  should  be  displayed  on  the  CRT  at  one  time,  considering  its 
readability;  (2)  the  complications,  manpower,  and  time  involved  in  preparing 
illustrations  for  use  on  the  image  projector,  as  mentioned  earlier,  and 
the  feasibility  of  using  graphics  on  the  CRT  when  considering  cost  in  time 
and  machine  efficiency  versus  the  educational  value  of  the  graphics.  The 
original  courses  underwent  Fairly  extensive  changes  to  help  overcome  these 
problems . 

A  report  was  compiled  by  the  Educational  Systems  Center* 
regarding  the  evaluation  of  the  CAI-1500  course  materials  as  they 


*5ummary  Validation  Report  on  the  CAI-1500  Courses  Developed  at  the 
U.  S.  Naval  Academy,  No.  PR-1070-35R,  Educational  Systems  Center, 
U.  S.  Naval  Academy,  Annapolis,  February  1971. 
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were  implemented  in  the  academic  year  1969-70.  Revisions  of  some 
form  were  made  to  all  of  the  course  materials  during  the  summer 
of  1970,  and  all  the  courses  were  implemented  at  least  once  during 
the  1970-71  academic  year.  Although  formative  evaluation  data  on 
course  performance  during  this  year  was  tabulated,  no  further  revisions 
were  made. 

The  formative  evaluation  process  involved  careful  examination 
of  all  performance  objectives;  the  validity,  reliability,  objective¬ 
relatedness,  and  comparability  of  test  items;  and  the  quality,  quantity, 
and  organization  of  content,  and  its  pertinence  to  the  defined  objectives.. 
This  required  scrutiny  of  curriculum  specifications,  the  conventional 
course  prospectus,  the  test,  the  CAI  objectives  and  programs,  student 
performance  data,  and  student  attitude  indexes. 

First,  CAI-related  objectives,  which  had  been  generated  by  course 
authors  and  approved  by  their  parent  academic  department,  were  reviewed 
to  assess  whether  they  were  (1)  sufficiently  inclusive  of  the  depth  and 
scope  of  the  learning  sought  in  the  course;  and  (2)  adequately  specific 
as  to  the  type  of  behavior,  the  degree  of  mastery,  and  the  conditions 
for  its  demonstration,  all  of  which  are  considered  essential  to  formulate 
fair  and  relevant  criterion  instruments  and  to  enable  optimal  selection 
and  use  of  strategies,  media,  and  content.  Those  individuals  designated 
as  course  evaluators  examined  the  objectives  defined  in  relation  to  the 
course  syllabus,  the  CAI  content  and  questions,  and  in  some  cases,  the 
text . 


Course  evaluators  next  studied  CAI  test  items.  They  reviewed  all 
program  questions,  most  of  which  had  reportedly  been  drawn  from  department 
pools  of  previously  validated  questions,  for  (1)  their  relevance  to 
objectives  and  to  the  instructional  materials;  (2)  their  apparent  com¬ 
parability  in  difficulty;  (3)  their  consonance  with  the  rules  for  sound 
question  construction;  and  (4)  student  performance  thereon.  The  questions 
pertinent  to  each  objective  were  evaluated  separately. 
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Expectedly,  the  types  of  questions  in  widest  use  were  found  to 
be  constructed  response,  true/false,  and  especially,  multiple-choice. 

They  were  further  categorized  according  to  their  utilization  within 
programs.  The  pre-test  item  intends  to  determine  the  level  of  student 
proficiency  prior  to  instruction  in  order  to  teach  to  his  needs.  The 
diagnostic  question  discerns  the  nature  of  a  student's  problem  so  that 
he  may  then  be  branched  to  remediation  tailored  to  meet  that  inadequacy. 

The  check-test  provides  drill  or  self-testing  and  is  properly  as  difficult 
as  the  pre-test  and  post-test  questions  since  it  should  reveal  to  the 
student  whether  he  has  mastered  the  objective.  Finally,  the  post-test 
item  measures  ultimate  achievement  of  the  terminal  obj ective- related 
behavior. 

Instructional  content  was  then  inspected  for  its  relevance  to 
the  behavioral  objectives,  sufficiency  for  achieving  the  desired  level 
of  learning,  and  its  application  of  accepted  principles  of  clarity, 
vividness,  logic,  difficulty,  etc.  Particular  attention  was  given  to 
the  remedial  sequences  which  were  aimed  at  specific  student  misunder¬ 
standings  . 

The  analysis  of  student  performance  data  was  especially  revealing 
in  those  courses  which  had  been  structured  to  allow  measurement  of  individu 
student  gain  between  pre-test  and  post-test  performance.  It  was  expected 
that  synthesis  of  individual  performance  data  would  pattern  out  the  major 
program  weaknesses,  and  further,  would  disclose  the  effectiveness  of 
materials  and  activities  such  as  review. 
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H.  EVALUATION  RESULTS 


Results  from  the  present  evaluation  of  the  CAI  materials  that  have 
been  prepared  and  used  at  the  Naval  Academy  lead  to  several  hypotheses 
concerning  CAI  and  the  state  of  the  art  of  CAI.  It  is  extremely  impor¬ 
tant,  however,  that  the  reader  remind  himself  that  the  present  evaluation 
is  in  reality  a  case-study  and  that  the  results  and  patterns  of  results 
which  emerged  may  be  limited  in  their  generality.  These  findings  simply 
show  what  resulted  when  CAI  was  employed  as  it  was  at  the  Naval  Academy. 
Definitive  statements  of  what  will  always  or  even  usually  happen  when  CAI 
is  used  can  be  made  only  when  results  of  other  CAI  projects  are  assessed 
and  compared  with  those  of  the  present  evaluation.  For  a  more  detailed 
description  of  the  evaluation  of  the  CAI-1500  materials,  see  Supplement  G 
(CAI-1500  Project  Evaluation  Report) . 

1.  TIME  SPENT  ON-LINE 

It  should  be  noted  that  the  fall  Chemistry  CAI  materials  were 
tutorial  in  nature  and  they  totally  replaced  two  weeks  of  classroom 
instruction  for  the  semester.  The  spring  Chemistry  materials  were  of 
the  drill-and-practice  mode,  and  they  were  used  in  lieu  of  conventional 
homework  on  the  topic  of  Kinetics.  Students  spent  a  range  of  4.2  to  14.9 
hours  on-line  with  the  fall  Chemistry  CAI  materials  and  from  .8  to  1.9  hours 
on  the  spring  Chemistry  CAI  materials.  The  Naval  Operations  Analysis  CAI 
materials,  like  those  of  spring  Chemistry,  were  drill-and-practice  in  nature, 
and  one  two-hour  class  session  every  two  weeks  was  allotted  for  practice  on 
these  homework  problems.  Students  spent  a  range  of  3.5  to  10.7  hours  on¬ 
line  with  the  NavOps  CAI  materials.  The  Russian  CAI  materials  were  also  of 
the  drill-and-practice  mode,  and  they  replaced  one  50-minute  class  session 
a  week.  Students  spent  a  range  of  5.8  to  23.8  hours  on-line  with  the 
Russian  CAI  materials. 
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2.  EXAMINATION  PERFORMANCE 


Control  groups  receiving  conventional  instruction  were  available 
for  comparison  with,  the  CAI  students  in  each  course  except  Physics. 

Statistical  comparisons  were  made  on  a  variety  of  combinations  of  hour 
exams  and  final  examinations  as  well  as  total  time  spent. 

Most  test  items  were  related  to  objectives  covered  by  the  CAI 
materials  (dual  treatment  items).  The  score  on  these  items  were  designated 
as  the  CAI-related  score  to  indicate  that  these  were  scores  on  which  a  direct 
comparison  could  be  made  between  the  CAI  treatment  group  and  the  conventionally 
taught  treatment  group.  The  other  test  items  (common  treatment  items)  were 
scored  and  made  up  the  non-CAI-related  score.  This  score  reflected  performance 
on  conventionally  taught  materials  for  both  treatment  groups.  Table  3 
indicates  the  number  of  students  in  the  CAI  and  conventional  treatment 
groups  for  each  course,  and  presents  comparative  percent  performance  on 
all  examinations.  There  was  one  course  where  the  performance  of  a  CAI 
group  was  significantly  more  favorable  than  that  of  the  corresponding 
control  group.  In  the  analysis  of  the  data  from  the  Russian  course,  the 
performance  of  the  CAI  students  on  three  of  the  exams  was  statistically 
significantly  better  than  that  of  the  control  students  recieving  con¬ 
ventional  instruction.  (Table  3.) 

3.  TIME  COMPARISONS 

While  there  were  no  overall  time  savings  through  the  use  of  CAI 
for  the  students  as  a  group,  the  instructors  were  enthusiastic  about  the 
time  savings  to  them  particularly  in  the  problem  solving  and  homework 
aspects  of  their  course.  They  appreciated  that  more  time  could  be  spent 
on  the  theoretical  aspects  of  their  instruction. 
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TABLE  3 


EXAMINATION  PERFORMANCE  BY  TREATMENT  GROUP 
FOR  THE  FOUR  CAI-1500  COURSES 


Treatment  Group 


CAI 

Conventional 

FALL  CHEMISTRY  -  Number  of  Students 

43 

49 

Hour  Exam  -  CAI  related  score 

75  9 

729 

Hour  Exam  -  Non-CAI  related  score 

56 

60 

Final  Exam  -  CAI  related  score 

75 

77 

Final  Exam  -  Non-CAI  related  score 

56 

57 

SPRING  CHEMISTRY  -  Number  of  Students 

49 

49 

Hour  Exam  -  CAI  related  score 

73 

74 

Hour  Exam  -  Non-CAI  related  score 

58 

60 

NAVAL  OPERATIONS  ANALYSIS 

FALL  SEMESTER  -  Number  of  Students 

34 

32 

Final  Exam  -  CAI  related  score 

75 

72 

Final  Exam  -  Non-CAI  related  score 

65 

65 

SPRING  SEMESTER  -  Number  of  Students 

61 

62 

Final  Exam  -  CAI  related  score 

59 

57 

Final  Exam  -  Non-CAI  related  score 

45 

46 

FALL  RUSSIAN  -  Number  of  Students 

9 

13 

Hour  Exams  Total  Score 

66b 

69b 

Pre- Final  -  CAI  related  score 

55 

45° 

Pre-Final  -  Non-CAI  related  score 

67 

69 

Final  Exam  Total  Score 

66 

60 

SPRING  RUSSIAN  -  Number  of  Students 

9 

8 

Hour  Exams  Total  -  CAI  related  score 

88C 

67C 

Hour  Exams  Total  -  Non-CAI  related  score 

80 

82 

Pre-Final  -  CAI  related  score 

39 

39 

Pre-Final  -  Non-CAI  related  score 

60 

80 

Final  Exam  Reading  Score 

40c 

39c 

Final  Exam  Listening  Score 

44 

38 

^All  exams  scores  are  given  as  percentages. 
Statistically  significant  difference:  p<.10. 
Statistically  significant  difference:  p<.05. 


NOTE:  NavOps  materials  used  were  the  same  in  fall  and  spring. 

Chemistry  and  Russian  (both  being  two-semester  courses) 
used  different  materials  in  fall  and  spring. 
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4.  STUDENT  ATTITUDES 


a.  Subject  Matter 

In  the  two  cases  where  attitudes  toward  subject  matter  were 
assessed  (NavOps  and  Russian) ,  it  was  indicated  that  the  use  of  CAI  may 
favorably  influence  the  attitudes  of  students  who  do  not  already  have 
relatively  favorable  attitudes  while  it  may  have  a  neutral  or  slightly 
adverse  effect  on  attitudes  of  students  with  very  positive  initial 
attitudes.  The  results  indicate  the  need  for  a  control  group  in  assessing 
changes  in  attitudes  toward  the  subject  matter.  In  both  Russian  and  NavOps, 
student  attitudes  toward  the  subject  declined  significantly  over  the  semester. 
However,  the  much  lesser  decline  experienced  by  the  CAI  students  might  be 
considered  an  improvement  in  student  attitude. 

b.  CAI 

Attitudes  toward  CAI  itself  were  significantly  more  positive 
for  CAI  students,  than  control  students  in  all  cases  except  those  for 
spring  Russian.  An  interesting  finding  relative  to  attitudes  toward  CAI 
was  that  the  CAI  students  invariably  showed  more  positive  attitudes  toward 
CAI  than  the  control  students  before  either  group  had  been  on-line.  While 
control  group  attitudes  remained  relatively  constant  across  the  course, 
the  initial  experience  on-line  caused  CAI  student  attitudes  to  become  even 
more  favorable.  This  increase  was  temporary  for  the  fall  and  spring  Chemistry 
CAI  students  since  their  attitudes  returned  to  their  initial  (but  positive) 
level  after  CAI  was  discontinued.  The  fall  and  spring  CAI  NavOps  students 
maintained  favorable  attitudes.  A  definite  decrease  in  favorability  occurred 
for  Russian  CAI  during  the  end  of  the  fall  semester  and  all  of  the  spring 
semester.  The  Russian  materials  were  more  time-consuming  and  assessed  to 
be  somewhat  more  difficult  for  the  students  than  the  other  course  materials. 
This  suggests  that  the  student's  attitude  toward  CAI  materials  may  reflect 
the  difficulty  the  student  has  with  the  materials  more  than  the  degree  to 
which  they  benefit  him  on  exams. 
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5.  INSTRUCTIONAL  DECISION  MODELS 


It  should  be  recognized  that  the  CAI  instructional  materials  used 
in  this  evaluation  went  through  several  revision  cycles.  However,  due  to 
"re-starts"  necessitated  by  curriculum  changes,  they  are  not  necessarily 
considered  perfect  teaching  packages.  It  is  felt  that  instructional  out¬ 
comes  could  be  improved  with  further  revision  cycles. 

Ii  the  same  respect,  with  insights  gained  from  previous  iterations, 
the  instructional  decision  models  could  also  be  improved.  The  instructional 
decision  models  employed  with  the  Naval  Academy  CAI  materials  were  generally 
unsophisticated  and  not  at  all  similar  to  the  Utopian  models  vaguely  outlined 
by  enthusiastic  proponents  of  CAI.  They  were,  instead,  simplistic  models 
employing  nobviousn  principles  (such  as  "do  not  teach  the  student  what  he 
already  knows").  However,  sophisticated  models  which  have  been  empirically 
proven  were  not  and  are  not  presently  available. 

The  decision  model  used  with  fall  Chemistry  was  appropriate  for 
a  majority  of  the  students,  but  it  allowed  a  sizable  group  of  students  who 
performed  well  on  pre-tests  to  bypass  instructional  materials  which,  judging 
from  post- test  performance  of  these  students,  were  sorely  needed. 

The  model  employed  with  the  Russian  materials,  while  effective 
in  terms  of  facilitating  examination  performance,  led  to  a  great  deal  of 
repetition  and  error  making  for  even  the  brighter  students.  Thus  a  distaste 
for  the  materials  developed,  as  evidenced  by  questionnaire  responses  and  time 
spent  on-line.  This  aversion  for  the  CAI  materials  would  not  be  expected  to 
be  conducive  to  educational  advancement.  None  of  the  decision  models 
incorporated  off-line  data  into  instructional  decision  making  and  none 
of  the  models  adequately  pointed  students  toward  off-line  assistance 
when  it  became  apparent  that  on-line  materials  were  not  producing 
effective  results.  The  individual  differences  analyses  indicated 
that  personality  and  attitudinal  scores  as  well  as  ability  test  data 
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should  be  consulted  in  making  instructional  decision?  and  assignments 
if  the  full  value  of  the  tutorial  and  drill-and-practice  modes  of  CAI 
is  to  be  demonstrated. 

6.  CAI  TECHNIQUE  AND  INDIVIDUAL  DIFFERENCES 

Two  techniques  using  the  CAI- 1500  system  were  adequately  tested: 

(1)  the  total-tutorial  mode,  wherein  all  necessary  instruction  is  given 
on-line,  and  (2)  the  drill-and-practice  mode,  wherein  basic  instruction 
occurs  in  class  while  skills  are  strengthened  through  use  of  CAI.  A 
third  mode,  adjunctive-tutorial ,  embodied  in  the  Physics  CAI  materials, 
but  class  enrollment  limitations  precluded  their  assessment  in  conjunction 
with  a  control  group. 

The  problem-solving  drill  application  of  CAI  as  used  with  NavOps , 
Russian,  and  spring  Chemistry  appear  to  be  most  beneficial  to  students 
who  would  be  expected  to  do  well  in  these  courses.  It  can  be  hypothesized 
that  the  drills  and  reviews  benefit  these  students  mainly  because  the 
students  are  more  likely  to  have  whatever  minimal  grasp  of  the  material 
is  necessary  to  benefit  from  drills  and  review.  Such  students  can  discover 
the  weak  points  in  their  knowledge  through  CAI  and  then  correct  their 
weaknesses  using  only  the  brief  hints  provided  on-line  or  their  own  resources. 
Students  who  lack  a  minimal  grasp  of  the  material  to  be  drilled  on-line 
received  relatively  little  assistance  on-line  and  thus  cannot  benefit  much 
from  the  CAI  materials.  It  would  appear  that  more  extensive  remediation 
should  be  provided  for  these  students. 

The  sole  case  of  total-tutorial  CAI,  fall  Chemistry,  seems  to  be 
most  successful,  relative  to  conventional  instruction,  with  students  who 
would  be  expected  to  experience  difficulty  in  learning  the  materials. 

Since  the  CAI  instruction  is  "self-contained"  and  requires  fewer  entry 
skills,  no  one  is  "lost"  at  the  start.  As  previously  mentioned,  the 
fall  Chemistry  deicsion  model  was  not  as  effective  with  all  students  scoring 
well  on  the  pre-tests  as  it  was  with  students  doing  poorly  on  pre-tests. 
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The  greatest  advantages  expected  from  CAI  in  general  are  due  to 
hypothesized  individualization  of  instruction.  CAI  would  ideally  tailor 
the  instruction  to  fit  each  student  on  the  basis  of  his  abilities, 
personality,  and  prior  knowledge  and  present  performance.  The  conditions 
of  conventional  instruction  at  the  Naval  Academy  probably  are  conducive 
to  "individualizing"  instruction  already.  Classes  are  small  ( 20  or  fewer 
students)  and  the  range  of  abilities  and  background  is  relatively  small 
within  each  class  due  to  admission  and  placement  procedures  at  the  Academy, 
Thus  the  instructor  can  pitch  his  presentations,  text,  and  reading  assign¬ 
ments  to  a  level  not  too  far  above  or  below  that  of  all  students  in  his 
class.  Since  the  instruction  situation  at  the  Naval  Academy  allows  for 
individualization  to  this  extent,  it  appears  that  if  CAI  materials  are 
to  be  significantly  more  effective  than  conventional  instruction  in  this 
setting,  they  must  employ  teaching  conditions  and  principles  superior  to 
those  in  general  use  at  the  Academy.  At  the  present  time  these  principles 
do  not  appear  to  be  adequately  and  reliably  established. 
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IV.  FINDINGS  AND  CONCLUSIONS 

A.  RELATIVE  TO  MAJOR  GOALS 

The  major  project  goals,  as  stated  in  Chapter  I,  were  to  evaluate 
the  use  of  the  computer  as  a  tool  in  performing  certain  instructional 
functions,  specifically: 

1.  INDIVIDUALIZING  INSTRUCTION 

a.  CAI  enabled  students  to  go  through  the  materials  at  their 

own  rate.  In  Russian,  for  example,  some  students  completed  the  instructional 
material  in  six  hours;  some  took  24  hours  to  go  through  the  same  material. 

b.  CAI  enabled  instructors  to  identify  individual  differences 
among  students  and  to  provide  remediation  for  those  who  needed  it. 

(However,  as  was  noted,  the  Naval  Academy  does  not  provide  the  same 
wide  range  of  individual  differences  as  would  be  found  in  most  civilian 
institutions  of  higher  learning.) 

2.  FACILITATING  SOLUTION  OF  COMPLEX  PROBLEMS 

The  teletype  gained  great  acceptance  by  both  faculty  and  students 
because  of  its  effective  utilization  in  solving  design  and  engineering 
problems.  It  was  particularly  effective  in  such  courses  as  Underwater 
Acoustics,  Fluid  Mechanics,  and  Physics. 

3.  DETERMINING  OPTIMUM  COURSE  CONTENT 

a.  The  development  of  CAI  course  materials  required  instructors 
to  be  precise  in  stating  behavioral  objectives  and,  consequently,  to 
re-evaluate  course  content.  This  was  particularly  true  in  Chemistry. 

b.  Some  courses  changed  due  to  changing  curriculum  requirements 
which  should  be  expected  in  any  college  environment.  This  caused  problems 
in  CAI  material  revision  and  data. 
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4.  RECORDING  AND  ANALYZING  STUDENT  PERFORMANCE  DATA 


a.  The  computer  aided  the  instructors  in  readily  obtaining  the 
data  which  indicated  where  students  were  having  difficulty  in  mastering 
material  and  where  course  revision  was  necessary. 

b.  The  methods  for  recording  and  analyzing  data  as  well  as  for 
producing  the  materials  and  managing  the  program  had  to  be  developed.  CAI 
theory  and  practice  was  at  a  primitive  stage  when  this  project  started. 

B.  RELATIVE  TO  STUDENT  ACHIEVEMENT,  ATTITUDES,  AND  TIME 

1.  Teletype  use  increased  the  amount  of  subject  matter  covered  in 
the  courses.  It  particularly  relieved  students  of  the  burden  of  large 
and  complex  manual  calculations  and  provided  more  time  to  understand 
principles . 

2.  Computer  use  in  the  tutorial  mode  to  supplement  conventional  instruction 
was  more  in  the  direction  of  increased  course  depth  and  quality  than  increased 
coverage,  decreased  cost,  or  relief  from  teaching  ftl ess  appealing"  parts  of 

the  course. 

3.  Teletype  use  was  more  successful  in  topics  of  greater  difficulty 
and  those  requiring  creative/intuitive  thinking  than  in  those  involving 
rote-learning. 

4.  The  teletype  was  an  extremely  powerful  teaching  tool  in  simulations, 
solutions  of  complex  problems,  laboratory  data  reduction  and  analysis. 

Eight  courses  were  soon  using  these  techniques  operationally.  It  was  less 
successful  when  using  drill  and  practice  and  tutorial  techniques.  Three 
courses  using  these  techniques  did  not  go  operational  but  were  continued  as 
research  projects.  Improved  student  achievement  was  not  demonstrated  using 
these  techniques. 

5.  In  the  1500  Project,  there  were  no  significant  differences  between 
the  performance  of  the  CAI  groups  and  the  control  groups  except  in  the 
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Russian  course  where  the  CAI  group  showed  significantly  better  achievement 
than  the  control  group. 

6.  In  the  1500  Project,  there  were  no  overall  time  savings  through  the 
use  of  CAI  for  the  students;  however,  the  instructors  were  relieved  of  many 
of  the  problem  solving  and  homework  aspects  of  the  course  and  could  devote 
this  time  to  other  instructional  duties. 

7.  Student  attitude  toward  the  tutorial  type  of  CAI  techniques  was 
generally  favorable,  reaching  a  peak  near  mid-semester.  Their  evaluation 
on  whether  or  not  the  CAI  support  had  helped  them  was  generally  positive. 

8.  The  more  closely  the  student  was  connected  to  the  computer  and 
personally  participated  in  the  programming,  (as  in  the  CAI-Teletype  Project), 
the  better  his  understanding  of  the  subject  matter. 

C.  RELATIVE  TO  CAI  TECHNIQUES 

1.  The  non-computational  techniques  (drill  and  practice,  tutorial, 
testing)  did  not  result  in  the  same  degree  of  acceptance  and  enthusiasm 
by  the  faculty  and  students  as  did  the  computational  techniques  (problem¬ 
solving,  simulation,  data-reduction  and  formatting).  These  techniques 
cannot  be  compared  directly  on  the  basis  of  student  achievement  or  time/ 
cost  savings. 

2.  Techniques  which  permitted  the  faculty  and  student  to  interact 
directly  with  the  computer  and  participate  by  writing  or  changing  the 
program  (CAI-Teletype)  were  more  widely  accepted  and  used  than  techniques 
requiring  a  technical  staff  to  do  the  programming  (CAI-1500).  Approximately 
50%  of  planned  academic  year  1971-72  connect  time  will  involve  the  use 

of  problem  solving,  and  over  20%  will  involve  simulation  techniques. 

Again,  these  two  techniques  cannot  be  compared  directly  on  a  basis  of 
student  achievement  or  time/cost  savings. 
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D.  RELATIVE  TO  UTILIZATION 


1.  Use  of  the  teletype  significantly  increased  student  participation 
in  the  subject  matter  of  the  courses. 

2.  The  use  of  teletype  expanded  very  rapidly  with  additional  faculty 
and  midshipmen  learning  to  write  programs.  The  number  of  leased  terminals 
installed  doubled  each  year  for  three  years.  Use  was  limited  by  operational 
funds  available  for  vendor  service  until  the  installation  of  a  large  time¬ 
sharing  system  (GE-635) . 

3.  Growth  of  teletype  use  was  Academy-wide  rather  than  by  academic 
discipline. 

4.  The  1500  system  was  not  configured  so  that  faculty  and  students 
could  easily  use  it. 

E.  RELATIVE  TO  COMPUTER  HARDWARE/SOFTWARE 

1.  The  original  enthusiasm  shown  by  faculty  and  students  diminished 
during  the  early  stages;  and  expanded  use  of  the  system  was  in  question 

for  a  time  until  both  the  teletype  and  1500  systems  were  made  highly  reliable. 

2.  The  1500  system’s  capability  to  randomly  access  large  video  and 
audio  files  and  to  record  data  made  it  an  excellent  instructional  research 
tool.  Where  configured  for  this  type  of  work,  however,  its  capacity  to 
handle  the  many  other  educational  computer  needs,  such  as  computational 
work,  was  severely  limited. 

3.  The  1500  system  was  limited  in  its  use  to  support  a  large  student 
load  due  to  the  maximum  of  32  terminals  that  can  be  used  simultaneously. 

4.  The  GE-635  proved  to  be  exceptionally  capable  in  handling  the 
computational  aspects  of  CAI  (problem  solving,  simulation,  and  data 
reduction  and  formatting) . 
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5.  Computer  language  for  the  1500  (COURSEWRITER  II)  is  essentially 
a  tutorial  type  language.  This  language  had  severe  limitations  in 
numerical  calculations,  string-processing  capabilities,  and  in  flexibility 
of  programming.  A  great  deal  of  effort  was  expended  in  training  personnel 
and  in  using  this  language  throughout  the  project. 

F.  RELATIVE  TO  STAFF  AND  SUPPORT  REQUIREMENTS 

1.  The  management  of  the  support  staff  needed  to  properly  prepare 
the  educational  material  for  use  and  to  collect  and  analyze  test  data 
consisted  of  a  diverse  group  of  experts  and  technicians,  whose  coordination 
resulted  in  a  highly  complex  operation. 

2.  The  collection,  management,  and  analysis  of  the  test  data  relative 
to  the  amount  of  course  material  prepared  required  resources  far  in  excess 
of  what  was  originally  anticipated. 

3.  Preparation  of  satisfactory  material  for  CAI  use  required  the  author 
to  have  experience  in  using  and  improving  his  material  repeatedly  during  its 
use  in  the  CAI  mode. 

4.  One  of  the  most  severe  problems  in  producing  and  revising  material 
for  the  CAI- 1500  Project  was  the  fact  that  most  authors  could  not  work 
directly  with  the  computer.  The  programmer  interface  was  an  ever-present 
communications  barrier  unless  the  author  either  used  clear,  detailed  Display 
Guides  or  learned  the  specialized  COURSEWRITER  II  computer  language. 

5.  Careful  planning  and  timely  production  of  teaching  aids  --  such 

as  art  work,  film  strips,  and  audio  tapes  to  support  the  course  material  — 
is  absolutely  necessary  to  the  project. 

6.  Faculty  changes  due  to  rotation  and  loss  affected  the  orderly 
execution  of  this  project  due  to  its  long  term  nature. 
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G.  RELATIVE  TO  COST 


1.  Use  of  the  teletype  for  computational  CAI  techniques  provides 

an  estimated  reduction  in  the  order  of  5-to-10-fold  for  solving  problems 
and  analyzing  laboratory  data  as  compared  with  "conventional  methods." 

Cost  of  vended  time  shared  service  ranges  from  $4  to  $6  per  terminal 
hour,  depending  upon  capability  and  degree  of  reliability  of  both  the 
remote  computer  and  communications.  Cost  may  be  as  low  as  $2  to  $3  per 
terminal  hour  for  an  installed  time-sharing  system.  A  student  in  one 
course  making  moderate  to  heavy  use  of  computers  (12  hours  per  semester) 
would  therefore  cost  from  $24  to  $72  per  semester.  Such  cost  can  be 
justified  and  considered  cost-effective  based  on  increased  course  level  -- 
the  incorporation  of  advanced  concepts,  techniques,  and  complex  problems. 

2.  The  use  of  a  dedicated  system  similar  to  the  IBM  1500  for  non- 
computational  CAI  techniques  is  extremely  expensive;  and  with  the  present 
state-of-the-art  of  CAI,  it  is  evident  that  there  are  many  alternate  ways 
of  providing  the  same  teaching  effectiveness  at  lower  operational  cost  such 
as  special  programmed  instruction  texts. 

3.  Factors  contributing  to  both  CAI-1500  and  Teletype  Project  costs, 
when  the  maximum  staff  was  employed  (1970-71): 

Average  Cost  Per  Year 


Hardware  (IBM  1500  System)  $  183,000 
Vended  Time-Sharing  Service  (Teletype)  60,000 
Management  Staff  81,000 
Programming  Staff  104,000 
Consultants  30,000 
Miscellaneous  (travel/supplies)  20,000 


Faculty  (no  summer  hire  during  this  period 
--  during  1968-69  was  $30-40,000 

per  year)  _ 

Total  Average  Cost  (1970-71)  $  478,000 
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4.  Development  time  for  the  CAI-1500  courses  varied  widely  and 
required  considerable  effort  to  revise.  The  development  time  per 
student  instruction  hour  using  the  IBM  1500  system  appears  so  high 
that,  unless  it  could  be  used  for  a  large  number  of  students  for  a 
long  period  of  time,  it  would  not  be  operationally  cost-effective. 
The  table  below  includes  man-hours  for  authoring,  programming,  and 
computer  operator  support. 


Initial 

Development  Re vis i on 

Total 

Development 
Time  Per  Hour 
Instruction 

Naval  Operations 

2483 

2543 

5026 

325 

Russian 

3675 

2894 

6569 

183 

Chemistry 

Data  Not  Available 

3810 

317 

Physics 

Data  Not  Available 

2545 

424 

5.  The  instructional  material  development  costs  can  be  reduced  by 
imposing  the  following  controls: 

a.  The  instructional  model  should  be  fixed  at  an  early  stage. 

b.  Inputs  by  behavioral  and  educational  technologist  must  be 
entered  at  a  specific  and  optimized  point  (and  as  early  as  possible). 

c.  Research  and  evaluation  design  features  must  be  realistic 
and  specific  at  the  start. 
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V.  RECOMMENDATIONS 


The  following  recommendations  are  addressed  to  those  educational 
institutions  (military  or  civilian)  that  may  be  contemplating  CAI 
applications  in  their  instructional  programs.  If  an  institution 
decides  to  become  involved  with  the  use  of  CAI,  the  following  recommendations, 
based  on  the  Naval  Academy fs  experience  and  the  present  state  of  the  art,  may 
be  helpful  in  minimizing  time  and  cost  while  maximizing  effectiveness. 

1.  Use  of  the  CAI-1500  system  for  non- computational  techniques 
(tutorial,  drill  and  practice,  testing)  was  not  operationally  or 
financially  effective  at  the  Naval  Academy.  It  is  recommended  that,  if 
a  dedicated  system  is  to  be  used  operationally  for  instruction,  the 
following  conditions  should  exist: 

o  Reliable  computer  system  is  mandatory. 

•  Large  numbers  of  students  are  necessary  to  reduce  cost 
per  student. 

•  Programs  should  be  simple  to  write  and  easy  to  change. 

•  Instructional  material  should  remain  reasonably  static 
so  that  frequent  changes  are  not  required. 

•  Considerable  effort  must  be  devoted  to  planning  the 
production  of  the  CAI  materials  and  the  data  management 
subsystem. 

•  The  system  must  be  capable,  not  only  of  providing  instruction, 
but  of  managing  data  on  student  performance. 

•  The  institution  must  have  personnel  resources  available  to 
provide  qualified  support  in  many  areas,  such  as  psychology, 
educational  technology,  and  computer  programming. 
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2.  From  a  cost-effective  viewpoint,  a  dedicated  CAI-1500  type  of 
system  is  not  recommended  in  a  university  environment  for  operational 
use.  More  hardware/software  system  development  is  required  to  reduce 
cost;  and  a  great  deal  of  educational  technology  research  must  be  done 

to  improve  development  of  instructional  materials  as  well  as  the  management 
of  student  learning.  The  few  courses  involved  and  the  relatively  small 
portion  of  each  course  committed  to  CAI  in  the  Naval  Academy  1500  Project 
emphasized  both  the  complexity  of  using  these  techniques  and  the  dearth 
of  existing  knowledge  on  how  they  can  be  used  successfully  in  a  college 
environment.  The  effort  that  appears  necessary  to  make  CAI-1500  type 
systems  operationally  feasible  is  of  such  magnitude  and  requires  expertise 
at  such  levels  that  it  should  be  accomplished  at  educational  institutions 
having  advanced  schools  of  education,  where  such  expertise  exists. 

3.  Use  of  the  CAI-teletype  system  for  computational  techniques 
(problem-solving,  simulation  data  reduction  and  formatting)  was  effective 
in  improving  the  academic  level  of  the  courses  involved  and  provided  a 
powerful  tool  using  techniques  unavailable  without  the  computer.  It  is 
recommended  that,  if  this  system  is  to  be  used  for  operational  instruction, 
the  following  conditions  exist: 

•  It  is  mandatory  that  the  computer  system  be  reliable. 

•  The  terminal  response  time  should  be  low  (1-3  seconds). 

•  A  simple,  conversational  computer  language  should  be  used. 

4.  On  the  basis  of  cost-effectiveness,  CAI-teletype  systems  are 
recommended  (and  are  being  used  by  the  military  for  educational  purposes 
in  most  of  the  service  academies  and  the  senior  service  schools).  The 
number  and  type  of  terminals  and  the  software  requirements  can  be  tailored 
and  easily  changed  to  meet  the  unique  requirements  of  each  institution, 
military  or  civilian. 


79 


APPENDIXES 


Page 

A.  Abstracts  of  Supplements  to  the  Final  Report  .  83 

B.  List  of  Members,  Naval  Academy  CAI  Advisory  Board  ....  97 

C.  List  of  Naval  Academy  Participants  in  the  CAI  Projects  .  .  101 


81 


APPENDIX  A 


ABSTRACTS  OF  ALL  SUPPLEMENTS  TO  THIS  REPORT 
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APPENDIX  A 


ABSTRACT  J  of  Supplement  A  - 

CAI -TELETYPE  PROJECT  REPORT 

This  supplement  documents  the  CAI -Teletype  Project  in  which 
standard  ASR-33/35  teletypewriters  were  employed  as  remote  terminals 
for  time-sharing  computing  systems.*  The  project  proceeded  through 
four  one-year  stages  which  have  been  labeled:  spot-testing,  imple¬ 
mentation,  validation,  and  evaluation.  During  the  first  stage  the 
participation  of  faculty  members  in  "spotting"  areas  of  potential 
application  was  encouraged  through  orientation  and  working  sessions 
to  develop  materials.  In  the  implementation  stage,  faculty  members 
used  the  CAI  materials  with  full  classes  of  students.  By  academic 
year  1967-68,  various  CAI  techniques  were  employed  in  12  different 
courses  and  in  a  special  project  that  provided  instruction  in  oper¬ 
ating  the  cathode-ray  oscilloscope.  The  validation  stage  took 
place  during  academic  year  1968-69  and  involved  the  critical  analy¬ 
sis  and  revision  of  four  courses:  Aerodynamics,  General  Physics, 
Electrical  Science,  and  Mechanics  of  Materials.  The  evaluation 
stage  consisted  of  the  experimental  evaluation  of  the  courses 
involved  in  the  validation  stage  with  the  exception  of  Aerodynamics. 
The  nature  of  each  course,  the  rationale  for  the  use  of  CAI,  the 
nature  of  the  CAI  materials,  and  the  experimental  conditions  and 
results  are  discussed.  In  addition,  an  analysis  of  attitudes, 
individual  differences,  and  problems  of  evaluation  is  provided. 

Beginning  with  a  summary  of  results  and  conclusions  pertaining 
to  the  various  types  of  CAI  examined  in  this  project,  the  final 
chapter  of  this  supplement  briefly  discusses  the  following  topics: 
current  status  of  CAI  at  the  Naval  Academy;  analysis  of  events  and 
factors  leading  to  the  growth  of  operational  CAI  at  the  Academy; 
trends  in  academic  computer  utilization  at  the  Academy;  and  overall 
conclusions  regarding  CAI-Teletype  costs,  benefits,  and  areas  of 
potential  payoff. 
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ABSTRACT 


of  Supplement  B  - 

DATA  MANAGEMENT  SUBSYSTEM  REPORT 


The  data  management  subsystem  (DMS)  developed  for 
support  of  the  Naval  Academy  CAI-1500  project  is  documented 
in  this  supplement.  The  main  body  of  this  document  concen¬ 
trates  on  the  analysis  of  and  an  approach  to  the  problem  of 
data  management  in  curriculum  development  and  educational 
research  projects. 

The  description  and  evaluation  of  the  Naval  Academy 
(DMS)  is  written  mainly  with  users  in  mind.  Documentation 
of  interest  to  computer  specialists  can  be  found  mainly  in 
the  manuals  and  programs  included  as  Appendixes. 

All  student  performance,  attitudes,  and  characteristic 
data  collected  during  the  evaluation  phase  of  the  CAI  pro¬ 
ject  are  on  file  at  the  Naval  Academy.  Data  file  structure 
and  record  formats  are  documented  in  the  Appendixes . 
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ABSTRACT 


of  Supplement  C  - 
PRODUCTION  OF  CAI  MATERIALS 


This  supplement  describes  the  approach  to  the  production 
of  CAI-1500  instructional  materials  which  evolved  over  the  course 
of  this  project. 

The  IBM  1500  Instructional  System  is  discussed  in  terms  of 
reliability,  limitations,  overall  costs,  and  modifications  made 
to  the  operating  system.  Topics  under  the  major  heading  of  man¬ 
power  include:  number  and  type  of  personnel,  time  required  for 
programming,  training  manuals,  and  general  staffing  considera¬ 
tions.  Problems  encountered  relating  to  manpower  are  discussed. 

The  procedures  for  programmer  assignment  and  the  establishment 
of  lines  of  authority  are  analyzed. 

Conclusions  are  presented  in  terms  of  general  development 
considerations  relating  to  costs,  supervision,  programming  stand¬ 
ards  ,  documentation  standards ,  manpower  requirements ,  and  computing 
systems  and  languages.  General  recommendations  are  offered. 
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ABSTRACT 


of  Supplement  D  - 
CAI  PROJECT  FACILITIES 


This  supplement  describes  and  assesses  the  facilities  used 
in  the  Naval  Academy's  CAI  project.  Standard  teletype  terminals 
accessing  remote  commercial  time-sharing  computers  were  employed  ini¬ 
tially  for  the  CAI-Teletype  Project.  In  support  of  other  computer 
uses  were  three  batch  processing  systems  at  the  Academy  (IBM  1620, 

IBM  7094,  and  IBM  360/30).  A  Honeywell  (GE)  635  time-sharing  system 
was  available  in-house  by  January  1971.  Space  for  this  equipment 
and  student  terminal  classrooms  is  discussed. 

The  1800-based  IBM  1500  Instructional  System  was  employed  in 
the  CAI-1500  Project.  The  components  of  this  system  are  defined  and 
their  configuration  is  delineated.  The  three  modes  of  use  (student, 
proctor,  and  author)  of  the  instructional  display  stations  are  described, 
with  attention  given  to  the  input  and  output  devices.  Requirements  for 
the  use  of  the  Time-Sharing  Executive  System  (TSX) ,  the  card/paper  tape 
programming  system,  FORTRAN,  and  the  subroutine  library  for  the  1800 
version  of  the  IBM  1500  system  are  outlined.  The  1500  programming 
systems  consisted  of  station  control,  station  command  processing,  CAI 
processing  (COURSEWRITER) ,  CAI  support,  and  CAI  utility  programs. 

The  operating  systems  available  for  the  Honeywell  (GE)  635 
computing  system  are  the  Dartmouth  (Phase  II)  Time-sharing  System 
(DTSS) ,  the  General  Electric  Comprehensive  Operating  System  (GECOS) , 
and  a  GECOS  simulator.  Some  of  the  languages  available  under  DTSS 
and  GECOS  are:  BASIC,  ALGOL,  LISP,  MIXAL,  FORTRAN  IV,  JOVIAL,  COBOL, 
GMAP,  and  SIMSCRIPT. 

An  assessment  of  the  suitability  of  the  facilities  used  in 
the  Naval  Academy's  CAI  project  is  provided.  The  general  performance, 
limitations,  and  costs  of  the  1500  and  635  computing  systems  as  well 
as  spocial  modifications  made  are  discussed.  General  recommendations 
relating  to  hardware  and  software  are  offered. 
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APPENDIX  A 


ABSTRACT 


of  Supplement  E  -  TECHNICAL 
RECOMMENDATIONS  BY  CONSULTANTS 
_ (2  volumes) _ 


This  supplement  consists  of  a  series  of  papers  by  two  consultants  -- 
Professors  Stolurow  and  Brennan  --  to  the  Naval  Academy's  CAI  and  Multi- 
Media  projects.  These  consultants  concentrated  on  issues  involved  in 
the  development  of  self-instructional  systems  and  evaluation.  However, 
owing  to  the  complex  nature  of  the  projects,  they  addressed  themselves 
as  well  to  problems  touching  on  nearly  all  aspects  of  the  Educational 
Systems  Center's  research  and  development  program,  particularly  in  the 
areas  of  statistics,  research  design,  and  CAI  as  well  as  behavioral  and 
educational  technology  in  general. 
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ABSTRACT 


of  Supplement  F  - 
FORMATIVE  EVALUATION 


This  supplement  documents  the  process  of  formative  evaluation  -- 
program  revision  based  on  student-generated  data.  Emphasis  is  directed 
toward  the  four  courses  developed  for  the  CAI-1500  Project.  These 
courses  were  basically  tutorial  and  drill-and-practicc  and  were  typically 
of  longer  duration  than  those  implemented  in  the  CAI-Tclctypc  Project. 

Course  development  and  revision  time  frames  arc  delineated,  and 
general  revision  and  data  collection  procedures  are  discussed.  The 
major  problems  encountered  in  the  formative  evaluation  of  the  four 
courses  are  outlined,  and  the  status  of  the  course  materials  prior 
to  the  last  implementation  is  given. 

A  brief  analysis  of  the  revision  procedures  employed  ’s  discussed 
in  terms  of  the  development  of  an  efficient  data  management  subsystem 
and  the  establishment  of  revision  criteria.  An  approach  to  objectify¬ 
ing  revision  procedures  is  introduced,  and  general  conclusions  and 
recommendations  are  offered. 

It  is  clear  that  the  formative  evaluation  process  is  necessary 
for  validating  individualized  instructional  materials.  It  is  also 
recognized  that  this  process  can  be  costly  and  time  consuming. 

It  is  essential,  therefore,  that  a  more  objective  approach  be 
developed  to  provide  decision  criteria  about  what  to  revise  when  a 
self-paced  instructional  program  fails  to  bring  about  the  desired 
level  of  performance. 

Further  work  should  be  done  to  clarify  the  field  of  criterion- 
referenced  testing  as  applied  to  instructional  programs.  In  addi¬ 
tion,  the  use  of  efficiency  measures,  latencies,  and  student  atti¬ 
tude  and  confidence  measures  as  variables  in  the  decision  process 
should  be  more  clearly  delineated. 
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Abstract-  of  SuPPlement  G  - 
n  \  CAI-1500  PROJECT  EVALUATION  REPORT 

This  supplement  treats  summative  and  research  analyses  conducted 
on  the  portions  of  the  four  courses  presented  via  the  IBM  1500  Instruc¬ 
tional  System.  The  basic  modes  of  CAI  employed  were  tutorial  and  dri^l 
and  practice.  However,  the  precise  application  of  each  mode  varied  m 
some  degree  from  course  to  course.  The  fall  Chemistry  and  Physics 
materials  were  tutorial  in  nature,  but  the  former  was  intended  to  com¬ 
pletely  replace  the  conventional  class  sessions  whereas  the  latter  was 
to  be  an  introduction  to  class  sessions  on  specific  topics.  While  the 
spring  Chemistry,  Naval  Operations  Analysis,  and  Russian  materials  all 
were  drill  and  practice,  they  varied  in  the  amount  and  type  of  remedia¬ 
tion  provided  as  well  as  in  the  specific  strategy  employed. 

The  types  of  analyses  discussed  for  each  of  the  courses  relate 
to  summative  evaluations  of  CAI  program  performance  and  comparative  evalu¬ 
ations  of  course  examinations,  study  time,  and  student  attitudes  toward 
both  course  content  and  the  medium  of  CAI.  Also  included  are  analyses  of 
individual  differences  which  identify  students  most  likely  to  benefit  on 
examinations  or  to  save  time  through  the  use  of  CAI. 

The  analyses  of  group  differences  (CAI  vs  conventional)  indicated 
that  the  CAI  materials  were  as  effective  as,  and  in  some  cases  superior 
to,  comparable  conventional  presentations  in  terms  of  instructional  out¬ 
comes  relating  to  examination  performance,  time  spent,  and  attitudes 
toward  content  and  the  medium  of  CAI.  Individual  difference  analyses 
indicated  that  students  with  certain  personality  characteristics,  atti¬ 
tudes,  interests,  and  cognitive  abilities  would  differentially  benefit 
from  the  use  of  CAI.  Instructional  decision  models  employed  with  the 
various  modes  of  CAI  should  be  sophisticated  enough  to  take  these  indi¬ 
vidual  differences  into  account  if  the  full  potential  of  CAI  is  to  be 
realized. 
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ABSTRACT 


of  Supplement  H  - 

CAI-1500  CHEMISTRY  COURSE  REPORT 


This  supplement  documents  the  development  and  evaluation 
of  the  General  Chemistry  material  implemented  via  the  IBM 
1500  Instructional  System.  Developmental  aspects  including 
organizational  and  manpower  requirements  from  the  initial 
efforts  in  1967  to  the  final  course  implementaion  in  1971 
are  discussed. 

The  instructional  hardware  and  software  utilized  are 
outlined  and  samples  of  the  instructional  materials  are 
provided.  Course  objectives,  flowcharts,  COURSEWRITER  II 
documentation,  and  student  comments  are  presented  in  the 
appendixes . 

The  evaluation  data  covers  the  last  implementation 
conducted  during  the  1970-71  academic  year.  CAI  program 
performance  and  performance  on  administrative  examinations 
as  well  as  time  and  student  attitudes  are  assessed.  Con¬ 
clusions  and  recommendations  are*  offered. 
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ABSTRACT 


of  Supplement  I  - 

CAI-1500  RUSSIAN  COURSE  REPORT 


This  supplement  documents  the  development  and  evaluation 
of  the  Basic  Russian  material  implemented  via  the  IBM  1500 
Instructional  System.  Developmental  aspects  including 
organizational  and  manpower  requirements  from  the  initial 
efforts  in  1968  to  the  final  course  implementation  in  1971 
are  discussed. 

The  instructional  hardware  and  software  utilized  are 
outlined  and  samples  of  the  instructional  materials  are 
provided.  Course  objectives,  COURSEWRITER  II  documentation, 
and  student  comments  are  presented  in  the  appendixes. 

The  evaluation  data  covers  the  last  implementation 
conducted  during  the  1970-71  academic  year.  CAI  program 
performance  and  performance  on  administrative  examinations 
as  well  as  time  and  student  attitudes  are  assessed.  Con¬ 
clusions  and  recommendations  are  offered. 
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ABSTRACT 


of  Supplement  J  - 
CAI-1500  NAVAL  OPERATIONS 
ANALYSIS  COURSE  REPORT 


This  supplement  documents  the  development  and  evaluation 
of  the  Naval  Operations  Analysis  material  implemented  via  the 
IBM  1500  Instructional  System.  Developmental  aspects  including 
organizational  and  manpower  requirements  from  the  initial 
efforts  in  1968  to  the  final  course  implementation  in  1971 
are  discussed. 

The  instructional  hardware  and  software  utilized  are 
outlined  and  samples  of  the  instructional  materials  are 
provided.  Course  objectives,  flowcharts,  COURSEWRITER  II 
documentation,  and  student  comments  are  presented  in  the 
appendixes . 

The  evaluation  data  covers  the  last  implementation 
conducted  during  the  1970-71  academic  year.  CAI  program 
performance  and  performance  on  administrative  examinations 
as  well  as  time  and  student  attitudes  are  assessed.  Con¬ 
clusions  and  recommendations  are  offered. 
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ABSTRACT 


of  Supplement  K  - 
CAI-1500  MODERN 
PHYSICS  COURSE  REPORT 


This  supplement  documents  the  development  and  evaluation 
of  the  Modern  Physics  material  implemented  via  the  IBM  1500 
Instructional  System.  Developmental  aspects  including 
organizational  and  manpower  requirements  from  the  initial 
efforts  in  1967  to  the  final  course  implementation  in  1971 
are  discussed. 

The  instructional  hardware  and  software  utilized  are 
outlined  and  smaples  of  the  instructional  materials  are 
provided.  Course  objectives,  COURSEWRITER  II  documentation, 
and  student  comments  are  presented  in  the  appendixes . 

The  evaluation  data  covers  the  last  implementation 
conducted  during  the  1970-71  academic  year.  CAI  program 
performance  and  performance  on  administrative  examinations 
as  well  as  time  and  student  attitudes  are  assessed.  Con¬ 
clusions  and  recommendations  are  offered. 
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LIST  OF  MEMBERS,  NAVAL  ACADEMY  CAI  ADVISORY  BOARD 
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CAE  ADVISORY  BOARD 


Captain  George  J.  Britncr,  USN 

Dir.,  Personnel  Research  Div.  (Pers  A3) 

Bureau  of  Naval  Personnel 

Department  of  the  Navy 

Washington,  D.  C.  20370 

(OX  2-0976) 

Dr.  John  J.  Collins 

Office  of  Chief  of  Naval  Operations 

Department  of  the  Navy  (OP  07TL) 

The  Pentagon  -  Room  511718 
Washington,  D.  C.  20350 
(OX  7-6509) 

Mr.  James  Duva 

Office  of  Naval  Research  (Code  55) 
Naval  Training  Device  Center 
Orlando,  Florida  32S13 


Dr.  Victor  Fields 

Office  of  Naval  Research  (Code  458) 
Department  of  the  Navy 
Washington,  D.  C.  20360 
(OX  6-3843) 

Mr.  Sidney  Friedman 
OUSN 

The  Pentagon  -  Room  4C731 
Washington,  D.  C.  20350 
(OX  7-6994) 

Mr.  Robert  F.  Graham 
NAVMAT 

Main  Navy  Building  -  Room  1121 
Washington,  D.  C.  20360 
(OX  6-7048) 

Dr.  H.  0.  Johnson 

Bureau  of  Naval  Personnel  (Pers  C3) 
Arlington  Annex  -  Room  4076 
Washington,  D.  C.  20370 


Dr.  J.  C.  Lang 

Bureau  of  Naval  Personnel  (Pers  Cd) 
Arlington  Annex  -  Room  3070 
Washington,  D.  C.  20370 
(OX  4-2796) 

Dr.  Richard  Ottc 
Bureau  of  Research 
Office  of  Education 
Room  3D011 

400  Maryland  Avenue,  S.W; 
Washington,  D.  C.  20202 
(963-4827) 

Mr.  Robert  Pruitt 
Bureau  of  Research 
Office  of  Education 
400  Maryland  Avenue,  S.W. 
Washington,  D.  C.  20202 


Mr.  Donald  Pollock,  Program  Dir-. 
Research  8  Studies 

Office  of  Scientific  Information  Svcs 
National  Science  Foundation 
Washington,  D.  C.  20550 
(632-5844) 

Dr.  Martin  F.  Wiskoff 

Bureau  of  Naval  Personnel  (Pers  A32) 

Arlington  Annex  -  Room  4611 

Washington,  D.'C.  20370 

(OX  2-09S0) 


-  99 


APPENDIX  B 


APPENDIX  C 
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CAI -TELETYPE  COURSES 

AND  INSTRUCTIONS 

Still  at 

Course 

Instructor 

USNA 

E210  Mechanics  of  Materials 

LCDR  D.  B.  Reynolds 
LCDR  B.  W.  Welles 

H .  B .  DeMart 

X 

E405  Fluid  Mechanics  II 

R.  F.  Latham 

X 

E408  Thermodynamics  II 

R.  F.  Latham 

X 

E631  Introduction  to  Aerodynamics 

R.  D.  Mathieu 

X 

E708  Heat  Transfer  I 

J .  A .  Adams 

X 

E710  Reactor  Physics 

W.  F.  Eckley 

X 

E732  Aerodynamics  I 

R.  D.  Mathieu 

X 

E735(L)  Aero  Performance 

CDR  D.  W.  Mathews 

E810  Reactor  Kinetics 

W.  F.  Eckley 

X 

E832  Aerodynamics  II 

D .  F .  Rogers 

C.  0.  Heller 

X 

E836  Aerospace  Design 

C.  0.  Heller 

D.  F.  Rogers 

X 

B.  H.  Carson 

X 

E924  Advanced  Topics  in 

C.  0  Heller 

Aerospace  Engineering 

V.  V.  Utgoff 

X 

N315/N316  NavOps  I  and  II 

P.  M.  Tullier 

N318  NavOps  (Demo  only) 

G .  B .  Hannah 

X 

S102  Chemistry 

LT  J.  L.  Koontz 

S211/212  General  Physics 

LCDR  John  Kropf 

LT  Anton  Vierling 

CAPT  John  Jones,  USMC 

X 

S305/306  Electrical  Science 

J.  L.  Daley 

C.  A.  Fowler 

X 

W.  K.  Kay 

X 

E.  J.  Eberhardt 

X 

S704  Computer  Aided 

R .  P .  Santoro 

X 

Underwater  Acoustics 

S.  A.  Elder 

X 

W931  Weapons  Systems 

D.  F.  Haber 

Oscilloscope  Operation 

A.  E.  Conord 

X 
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CAI-1500  COURSES  AND  INSTRUCTORS 


Course 


Instructor 


Still  at 
USNA 


XS206)  SP301  Modern  Physics 


E.  A.  Hall 
LCDR  J.  Kropf 
D.  A.  Nordling 

F.  L.  Miller 


FR101/FR102  Basic  Russian 


*C.  P.  Lemieux 
K.  E.  Lappin 
H.  R.  Keller 


NA311  Naval  Operations  Analysis 


*CDR  R.  M.  Olsen 
*LCDR  D.  J.  Kenney 
*LCDR  C.  E.  Peterson 
G .  B .  Hannah 
LTJG  T.  L.  Sopwith 
LT  J.  H.  Reed 


SC114  General  Chemistry 


Miscellaneous  and  Special  Projects 


LT  P.  A.  Boudreau 
LTJG  W.  L.  Mohr 
*ENS  J.  B.  Flanegan 
*J.  H.  Klein 
*W.  M.  Smedley 
*MAJ  R.  F.  Larriva,  USMC 
*CDR  R.  W.  Sirch 
LTJG  M.  Deverell 
LT  E.  Dabich 
LCDR  Patrick  McKinnon 
LT  David  McCaffrey 

LCDR  Markham  Tuft 
W.  Geatches 
R.  Driftmeyer 
K.  F.  Read 
Chih  Wu 
W.  M.  Lee 
MAJ  R.  Gulick 

D.  F.  Haber 

E.  E.  Betz 


*Administrator,  not  author 
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